try ls
[liveuser@localhost ~]$
It actually tells us
In this case, our username is "liveuser" and we are on a machine called "localhost" and we are currently in our own home directory and we have no special priviledges that will allow us to trash the system.
The magic decoder ring is broken so let’s do this the manual way:
The entire string is called a "prompt". It has a WHO@WHAT format for username and hostname. The WHERE, the ~ in this case is seperated by a space from the hostname. All of that is enclosed in the [] square brackets. The $ at the very end tells us we are a normal user and not allowed to change things for others.
delimiter A delimiter is used to seperate fields in a string of text. The prompt uses a @ and a space. The default delimiter is whitespace which is any character that is essentially blank like a space or a tab. |
How do I tell who I am? Or what machine I’m on. Or where I am?
Simple! Just type:
whoami
hostname
pwd
$HOME
?First we need to look around and see.
The ls
command will list the contents of the current directory.
Many modern systems will list contents color coded for easy identification of files from folders (It uses an alias
to do this). It will also show much more.It’s the workhorse for most users. It has many flags that modify it’s output. Some particularly useful ones are
try ls
Look at the difference between ls and ls -la in your home directory. Those listings that start with a . (dot) are "hidden". They are most often used as configuration files for user level applications.
Try running ls -lart and see how the files are rearranged. Lastly, run ls -Z /tmp. The section that has text separated by : are called SELinux file contexts.
um. yeah. the first character of the name is a dot.
duh!
They are often found in home directories and /tmp and typically store configuration data.
b4d h4k3Rz A host that has been broken into will often have spurious dot files with 3 dot names or names that are all spaces. |
[root@hackedbox badguys]$ ls -la
total 0
-rw-rw-r--. 1 root root 0 Feb 17 17:42
drwxrwxr-x. 4 root root 100 Feb 17 17:43 .
drwxrwxrwt. 12 root root 280 Feb 17 17:42 ..
drwxrwxr-x. 2 root root 40 Feb 17 17:43 ..
drwxrwxr-x. 2 root root 40 Feb 17 17:43 ...
echo will take the contents of a variable and output it to STDOUT. Use this to find out what’s in a variable echo $HOME where $HOME is a user environment variable (see your user environment with the env command.). Use echo to quickly create text files with echo "Hello, World" > foo. Also, you can look at the absolute path to your home directory with echo $HOME
echo $HOME
echo "Hello, World"
echo $PATH
cat is short for concatenate. It is used to join 2 or more files in sequence and display the contents to the screen (STDOUT).
cat /etc/filesystems /etc/shells
will display /etc/filesystems followed immediately by /etc/shells. By using IO redirection the contents can be sent to a file or into a string of other processes.
cat /etc/shells /etc/filesystems > /tmp/catfile
will create a file /tmp/catfile and put /etc/shells and /etc/filesystems into it.
cat shells | grep "sbin"
will take the output of the cat and use it as the input for the grep searching for "sbin". This is an overly complicated process as grep "sbin" /etc/shells is more streamlined and appropriate but it has the same effect. Type the command
cat /etc/filesystems - >> /tmp/foo.
When you get the blank line, start entering text. A new line is given when you press "Enter". To stop entering text press Ctrl-d
( Ctrl-d
is a EOF, or End Of File ). You should have a file /tmp/foo that has the contents of the /etc/filesystems followed by the lines you entered. The secret to the lines you entered was the -
. That is a shortcut that means "get the data from the keyboard"
To read your file, type cat /tmp/foo
.cat is not a reader
There are other ways to display the contents of a file. cat will do it but that’s not what it’s purpose is. |
cp will create a copy of a file or files. It have a multitude of flags that also allow keeping intact the file permissions, timestamps and many other things.
cp -a /etc/filesystems ~
Look at it with ls -l and cat. Now try and copy it again to the same location but use the -i flag instead of the -a flag. You will have to enter a "y" to actually copy it. Again look at the file with ls -l and note what changed.
This is used to move a file from one place to another as well as to rename a file.
mv /tmp/foo ~
will move the file foo from /tmp to the users home directory. The original file /tmp/foo is no longer in /tmp.
mv foo.txt foo.bak
will change the name of the file foo.txt to foo.bak. The mv command requires the user have rw permissions on both end of the process. Create a file called file1 with the command:
touch file1
Rename file1 to myfile.txt
Now move that file to /tmp
Lastly, move the file from /tmp back to your home directory while changing its name to foo.txt
mkdir is used to make directories:
mkdir dir
Using mkdir with -p will create all parent directories if they do not exist. Without -p, creating a directory inside a non-existing directory will fail.
mkdir -p dir1/dir2/dir3/
mkdir / rmdir The opposite of mkdir is rmdir |
mkdir /tmp/test
and ls /tmp/test
to verify
Now mkdir /tmp/test/foo/bar
and notice the failure message.
Make it work with the -p flag and verify with ls
Remove the directory /tmp/test.
Use this to change directories. Either enter the directory with a full path
cd /etc/sysconfig/network-scripts
or use the dot notation for directories above
cd ../../etc/yum.repos.d
where the ".." means "go up one level". A single "." means "right here in this directory".
cd [enter]
will jump the user’s home directory as will cd
(The is a shortcut for the users home directory).
cd -
will go back the previous directory the user was in (This is stored in a variable called OLDPWD. Type echo $OLDPWD
to see it)
cd /tmp and look around with ls
Now cd / and look around.
Use the shortcut to go back to the /tmp directory.
Use the shortcut to go back to your home directory.
Where am I?
pwd will show you the present working directory.
tape archive has more flags than can be covered in a week. It was originally used for copying data to and from tape drives (it is still used for that). It is similar to zip in that it can take a bunch of files and make a single archive file and compress it. The file created by the tar command is often called a "tarball".
tar cvzf file.tar.gz foo/ bar/ baz*
will create a new archive file called file.tar.gz while showing what is being done verbosely and compressing with zip while sourcing from the directories foo/ and bar/ and any file beginning with baz in the current directory. This is a major workhorse function along with it’s untar sister:
tar xvzf file.tar.gz
will unzip the file file.tar.gz being verbose and extract to the current directory the contents of the archive.
tar tvzf file.tar.gz
will tell the contents of a gzipped tarball (useful to use before the tar xvzf to make sure the archive has it’s own directory and doesn’t just plop a ton of files in the current directory!) tar xvjf file.tar.bz2:: will unpackage a tarball compressed with bzip2 compression. Note the changed flag and the archive ending. tar xvjf file.tar.gz -C newdir :: will untar file.tar.gz after changing directories into newdir. tar xvjf file.tar.bz2 foo.txt:: will extract only the file foo.txt from the tarball.
Your turn:
Create a tarball of the /etc directory in the /tmp directory.
Extract only the /etc/sysconfig folder from the tarball into the new directory /tmp/newdir
This will create an empty file with the given filename or update the timestamp on an existing file.
touch /tmp/testfile
and read the timestamp with ls
.
Use echo to add some text to the file and recheck the timestamp.
The timestamp you see is the "last modified" time (mtime).
Timestamps There are other times tracked by the system such as creation time ctime and access time atime .These can be seen with ls -l --time=STYLE where STYLE is ctime or atime. |
touch /tmp/testfile again and read the timestamp with ls.
cat /tmp/testfile and check the atime timestamp.
This will extract part of a line as split up by singe character delimiters
cut -f 3 -d ":" </etc/passwd
will output a list of UID’s. Cut will extract fields from a string or lines from a file using a defined single character delimiter. If -d is not specified, the default delimiter is the tab
character.
Extract a list of group names and GIDs from /etc/group
The history command will show the last (bazillion but closer to 1000) commands from the current session. This is an immensely useful tool. It’s a built in part of most shells and bash history is quite good. The storage place is in ~/.bash_ history. From the bash man pages:
history [n] history -c history -d offset history -anrw [filename] history -p arg [arg ...] history -s arg [arg ...] With no options, display the command {{history}} list with line numbers. Lines listed with a * have been modified. An argument of n lists only the last n lines. If the shell variable HISTTIMEFORMAT is set and not null, it is used as a format string for strftime(3) to display the time stamp associated with each displayed history entry. No intervening blank is printed between the formatted time stamp and the history line. If filename is supplied, it is used as the name of the history file; if not, the value of HISTFILE is used. Options, if supplied, have the following meanings: -c:: Clear the history list by deleting all the entries. -d offset:: Delete the history entry at position offset. -a:: Append the ''new'' history lines (history lines entered since the beginning of the current bash session) to the history file. -n:: Read the history lines not already read from the history file into the current history list. These are lines appended to the history file since the beginning of the current bash session. -r:: Read the contents of the history file and use them as the current history. -w:: Write the current history to the history file, overwriting the current history file contents. -p:: Perform history substitution on the following args and display the result on the standard output. Does not store the results in the history list. Each arg must be quoted to disable normal history expansion. -s:: Store the args in the history list as a single entry. The last command in the history list is removed before the args are added. If the HISTTIMEFORMAT variable is set, the time stamp information associated with each history entry is written to the history file. The return value is 0 unless an invalid option is encountered, an error occurs while reading or writing the history file, an invalid offset is supplied as an argument to -d, or the history expansion supplied as an argument to -p fails.
Uses of history The history command is very useful when building shell scripts to test out the pieces and pull up a record of what was done. |
Write the history of your current shell session to /tmp/hist using history -a /tmp/hist
Store the text fred flintstone in your history using history -s fred flintstone
Check the last 10 entries of your history using history -n 10
It does just what it sounds like it does, it finds files.
Files Everything in Linux is a file |
In general the command is run in the form find <path> <filters>
. Some very important ways to use find include searching for files using timestamp information or user or group ownership or with particular permissions or of a particular size or file type or having a particular SELinux context. The output of the find command is a filename with the full path. This can be fed into tools like xargs for further processing. find makes Microsoft search function look like a child’s toy.
Finds all files with "Foo" in name from current directory and subdirectories
Finds all files with "foo" in the name with no case sensitivity: finds foo, Foo FoO, etc
Finds all files named passwd that have at least 2 "/"s in their path
Finds all files named passwd with no more than 3 "/"s in their path
Find all files of type TYPE where TYPE is
Find all hidden files
Find files changed more than 2*24 hours ago
Find files changed more than 15 minutes ago
Find files change less than 15 minutes ago
Find files access less than 24 hours ago
Find the hidden files in your home directory Find the hidden files in your home directory than have been modified in the last 48 hours Find the hidden directories in your home directory that have been access less than 3 days ago
xargs is a great utility for executing a command on a list of input strings, etc. Paired with find, for example, xargs can copy, move, or remove matched files.
By default, xargs will pass all input lines as individual parameters to one call of the specified program. With the -I {} option, xargs will invoke the specified program once per line, and replace the string {} with the input line. This is useful for programs that can only deal with one filename at a time, or where the input string cannot be the final parameter of the command.
First create a test directory: mkdir test
Then cd into it and create three test files with touch foo bar baz
.
Then run find . -name "b*"
, examine the output, and ensure you understand the find invocation.
Finally run the same find command, followed by | xargs -I {} rm {}
.
When you are done, cd ..
and rm -r test/
to remove the test directory.
pipe | The |, or pipe symbol is used to transfer the output of one command to the input of another. |
which command name
whereis command name
locate filename
file filename
ps -aux
*Report process status
kill -9 pid
pid
. kill -9 will kill the process even if it can’t shut down gracefully.
df
du -sh filename
free -h
top
head -n N filename
tail -n N filename
ln -s filename1 filename2
rm -rf *
Test First! Test a recursive rm -rf * by substituting ls -R * first. |
echo "new stuff" > oldfile
Use correct redirect Check the redirect before hitting [enter] and see if the append form, ">> ", is more appropriate. |
/
|-- bin -> usr/bin
|-- boot
|-- dev
|-- etc
|-- home
|-- lib -> usr/lib
|-- lib64 -> usr/lib64
|-- lost+found
|-- media
|-- mnt
|-- opt
|-- proc
|-- root
|-- run
|-- sbin -> usr/sbin
|-- srv
|-- sys
|-- tmp
|-- usr
`-- var
Special directories are /dev, /proc and /sys as they provide data about the physical and virtual devices, running kernel processes, and userspace objects like devices and busses and loaded modules.
/boot
is used during bootup and stores the running kernel, initial run time filesystem and the bootup controller called "grub".
/etc
is used to store (almost) all the configuration files for the entire system. /etc/sysconfig
is for RedHat/Fedora specific system configurations.
/lib
and /lib64
hold common libraries for the system and applications.
/tmp
is a special holding place for temporary data, sockets used by user processes and many other non-permanent things. It is typically cleared out during a system reboot.
/root
is the home directory for the user root.
/var
holds system data that is either changing (log files and certain application data) or is served to the network (web pages, email, etc).
/home
is where user home directories typically are placed.
/media
is a typical mount point for user inserted media like DVD and thumbdrives.
/bin
is the location for most user runable binaries. It’s huge.
/sbin
is used to store system binaries that are not available to non-admin users.
Windows filetree is device driven. So if you add a new hard drive, your CD/DVD drive is now drive E: and no longer D: and your box may no longer boot up.
Linux uses a file (GASP!) to store where everything gets mounted called /etc/fstab
(filesystem table). They look similar to this one:
# # /etc/fstab # Created by anaconda on Wed Sep 11 21:00:50 2013 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/fedora_dhcp061164-root / ext4 defaults 1 1 UUID=f4fa206c-1342-459a-b2bc-38c1b1059692 /boot ext4 defaults 1 2 UUID=5e7d30d1-2bfe-3102-8f5b-b2811f089e58 /boot/efi hfsplus defaults 0 2 /dev/mapper/fedora_dhcp061164-home /home ext4 defaults 1 2 /dev/mapper/fedora_dhcp061164-swap swap swap defaults 0 0
Each line is composed of 6 parts. Source, mount point, filesystem type, mount options, filesystem dump flag and lastly filesystem check order.
The mount option defaults means:
rw - Mount the file system read-write.
suid - Allow set-user-identifier or set-group-identifier bits to take effect.
dev - Interpret character or block special devices on the file system.
exec - Permit execution of binaries.
auto - Can be mounted with the -a option (i.e. automatically at boot time or manually with the -a)
nouser - Forbid an ordinary (i.e., non-root) user to mount the file system. This is the default.
async - All I/O to the file system should be done asynchronously.
mount command The mount command lists all the currently mounted devices and ther mountpoint. |
older format that did not support a rapid filesystem recover process called journalling.
modern format that supports journalling by tracking open files with a write log journal that is written to first, then the actual file is written to and the journal entry is deleted. Very useful if the system goes down without first performing a write buffer flush operation.
new replacement for ext3 that is faster and solves some issues with ext3. Both ext3 and ext4 use journals but ext4 has greatly expanded filesystem size limits (exabytes) and file sizes (16 terabytes). See http://en.wikipedia.org/wiki/Ext4 for a good overview of ext4 improvements over ext3.
swap is special filesystem space used to park memory that has not been used in a while when the kernel needs more memory than it has available. It is similar to "virtual RAM" in that is uses the much slower hard drive space. When a server is thrashing it is constantly swapping memory back and forth from swap space. The system gets slower and slower until an overly large process is killed off and the swap space is no longer in heavy use. NOTE: if there is a drive error in the swap space partition and the kernel tries to access that data, the system will typically crash.
There are 3 versions of the Network File System: nfs, nfsv3 and nfsv4. The ancient nfs should never be used as it’s totally insecure and very buggy. nfsv3 addressed the security issues but left the instability issues. nfsv4 is both fairly securable and quite stable. With nfs, a file server can provide a common data storage area across a LAN (or WAN with v4 and vpn tunneling).
Modern format for most optical drive media. Usefull when mounting an iso file as a physical disk.
ll /dev/[cd][dv][drw]* lrwxrwxrwx. 1 root root 3 Feb 5 09:41 /dev/cdrom -> sr0
ll
is an alias for ls -l
in Fedora and RedHat systems.
Regular expressions was used in [cd] to choose one or the other character and again in [dv] and [drw]. +The end * means anything in any quantity.
RegEx Whole books have been written on regular expressions. They are slightly different for each operating system platform but still mostly similar enough. Get a book or Google up a RegEx cheatsheet. |
$ mount | grep boot /dev/sda3 on /boot type ext4 (rw,relatime,seclabel,data=ordered) $ grep boot /etc/fstab UUID=f4fa206c-1342-459a-b2bc-38c1b1059692 /boot ext4 defaults 1 2
The time
command will output how long a process runs to STDERR by default. Time how long a find runs looking for the find command and output all to a file.
find / -name find find: ‘/root/.ssh’: Permission denied find: ‘/root/.config’: Permission denied find: ‘/root/.pki’: Permission denied find: ‘/root/.dbus’: Permission denied find: ‘/root/.fltk’: Permission denied find: ‘/run/udisks2’: Permission denied find: ‘/run/gdm’: Permission denied find: ‘/run/libvirt/network’: Permission denied <snip>
Ah, HA! Remember that commands are in /sbin or /bin which are linked to /usr/sbin and /usr/bin
find /usr/*bin -name find
/usr/bin/find
Excellent! Now how long does it take?
time find /usr/*bin -name find /usr/bin/find real 0m0.012s user 0m0.004s sys 0m0.007s
This is really sneaky so I’ll cut to the chase
$ { time find /usr/*bin -name find >/tmp/timetest; } 2>>/tmp/timetest $ cat /tmp/timetest /usr/bin/find real 0m0.011s user 0m0.006s sys 0m0.006s
bash shell has a time command that fires off before the system time command does. To capture the STDERR of a process that hasn’t finished (bash is still running) wrap the entire command in { } and put the final STDERR to file on the outside. The secret sauce is the ";" that tells bash "that’s it. Now run it".
OK. Once more using the system time command and not bash
$ which time /usr/bin/time $ $(which time) find /usr/*bin -name find >/tmp/timetest 2>>/tmp/timetest $ cat /tmp/timetest /usr/bin/find 0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1836maxresident)k 0inputs+0outputs (0major+605minor)pagefaults 0swaps
The "$(which time)" is a bash subshell and it returns "/usr/bin/time" which is the full path to the command which then executes. Note the "2>>/tmp/timetest" is AFTER the first redirect so the inner portion can complete first. Redirecting the STDOUT to a file doesn’t require the full 1>[filename] notation as STDOUT is the default.
Linux has multiple text editors. pico and nano are rather tools. EMACS is huge and nearly it’s own operating system. vi, or it’s more modern version, vim, are always installed so admins can edit files with limited resources. vi is a text editor. It is not an email client. It is not a web browser. It won’t use a synthetic voice to read you your email. It will take a bit of time to learn how to use it effectively but that will pay off greatly.
vim works in 2 modes: insert or command. [esc]
will put it into command mode. "i" puts it into insert mode.
[shift]-A
moves to end of current line AND goes into insert mode
:w newfile
will write the current file to a new name newfile
vim supports syntax highlighting for nearly every programming language. To learn more, type vimtutor and follow the builtin tutorial.
If you run the following command, you’ll get a count of how many binaries there are in /usr/bin
ls /usr/bin | wc -l
On my system I get 3074. I don’t have everything installed that I usually do so that’s a little smaller than normal.
How to keep track of all of those and all the flags and options for each one requires decades of experience or
man is short for manual.
man ls
man rm
man echo
Head Exploding man gcc may cause uncontrolled physical spasms and possible loss of bodily fluids at more than 12000 lines. |
secure shell uses encryption to connect between systems. Has ability to provide port tunnelling. Highly secure with keys.
secure copy can move files from machine A to machine B while using machine C through encrypted tunnels. Uses ssh connection protocols.
remote sync can use an ssh tunnel to replicate files or entire filetrees.
ssh user@remotehost
ssh user@remotehost 'command1; command2'
ssh -X user@remotehost '<X command>'
ssh -L <localport>:localhost:<remote port> <username>@<hostname>
user user@ not required if the same user ID on both machines |
Requires public/private key combo.
Private key stays secret and public key gets installed on remote machines.
ssh-keygen
ssh-copy-id
scp file1 file2 host2:
scp user@host1:path/to/file user@host2:/path/to/destination
scp -r somedir host2:
Path default path begins with $HOME on remote machine. If outside of $HOME use full path from /. |
rsync -av workdir/ backupdir +
rsync -avz -e "ssh -l <remoteuser>" \
<remoteuser>@<remote host>:/path/to/file local/path/to/file +
rsync -avz -e "ssh -l <remoteuser>" \
<remoteuser>@<remotehost>:'/path/to/file1 /path/to/file2' localdir
Deletes rsync supports removing files from the destination if not present in the source. Be careful with this. |
Almighty Admin can rebuild failed hard drives manually in hex
nothing. worm. lower than dirt.
You can only break your stuff.
ls -la /home
ls -la /home
drwxr-xr-x. 4 root root 4096 Aug 7 2013 .
drwxr-xr-x. 18 root root 4096 Feb 17 18:24 ..
drwx--x---+ 37 jkinney jkinney 4096 Feb 18 23:14 jkinney
d
|
directory |
rwx
|
owner can read, write and execute |
r-x
|
group can read and execute |
r-x
|
world can read and execute |
.
|
There is an SELinux access control applied |
SELinux SELinux is like firewalling processes on the machine. Very solid (and often confusing) system security. SELinux rules are considered "Mandatory Access Controls" and can’t be bypassed without special permissions. |
d
|
directory |
rwx
|
owner can read, write and execute |
--x
|
group can execute |
---
|
world has no access |
+
|
There is a general access control applied (ACL) |
ACLs ACLs predate SELinux. They are similar but can be set by users on their own files. They are "Discretionary Access Controls". There are "default ACLs" that can exist on directories and those set the loosest ACLs a user can change to. |
ACLs, SELinux and tar don’t mix tar doesn’t backup (and thus can’t restore) the ACLs or SELinux context data. Use star instead. |
sed, awk, grep, diff, patch, chmod, chown, tr, split, join, df, du, wget, ncat, tac, lsmod, lspci, lsusb, seq, sudo, tee, zgrep, which, locate, su
These three are workhorses
Every command at the prompt can be used in a shell script.
ls /bin/f* | wc -l
The commandline is easy. Let’s write a script using vi.
vi countbin
press i to get into insert mode and type:
#!/bin/bash # This will output a count of files beginging with 'f' in /bin ls /bin/f* | wc -l
press [esc] to get into command mode and save and close with :wq
List the directory
Now make it executable with chmod 755 countbin
Run it with
./countbin
for i in $(seq 1 10); do mkdir testdir${i}; done
for loop iterates. while loop uses logic test
export x=1
while [ $x -le 5 ]; do echo "Welcome $x times"; x=$(( $x + 1 )); done
x=3 echo $x unset x echo $x
mydir='/home' myfile=${mydir}'/newfile' echo $myfile
which vi
echo $(ls -la $(which vi))
This presentation was written using a command line tool called asciidoc using vim.
It was "compiled" with the command
asciidoc --backend slidy --attribute duration=60 Linux-cli.adoc