Tuesday, July 28, 2009

WTF Is Linux?

Well, I am here to tell you, but before we launch into an explanation of what Linux is, we need to cover a few basics. First, I want to tell you a little about computer hardware, and then a little about how an operating system is structured.

On a typical desk in the USA you will see a computer. Sometimes this machine is a laptop computer, other times it is a desktop, and occasionally it is a palm top computer. No matter the type of machine it may be, it is still a computation machine (computer). Computers have existed for thousands of years, and only recently became electronic. The way we know them is new to the world. Computers as we know them were not invented until the 1970s. These types are called microcomputers, or personal computers; they are fairly uniform in nature, and have several common components that perform various tasks.

First, we have the box, the monitor, the keyboard, the mouse. That gives you one output device (monitor), and two input (mouse, keyboard). On the box you may have a floppy drive, zip, drive, cdrom drive, dvdrom drive, and so on. Those would be considered storage media, more to the point removable storage media. Inside of the box, there are more components. You have a power supply, normally some fans, a motherboard, a hard disk, some RAM, and sometimes some removable expansion cards (on pci, pcie, agp, or isa). Then underneath of a rather large cooling system, you have your processor. There are many more components but we needn't get into too much detail about all of this.

The power supply converts current coming from your wall into voltages that will not harm the rather sensitive components that make up your machine. It then sends this power to the internal components.

Many people get confused when a technologically inclined individual starts talking about RAM, processors, and hard disk drives. These are not hard concepts though. Your RAM is like desk space. The more you have the more you can work on at once assuming the desk never gets larger than the reach of your eyes and arms. Behind or beside your desk you may have a filing cabinet. This cabinet would serve as a hard disk drive. You would be the processor in this case. If your machine is either a) antiquated or b) very new, you may have coprocessor. This means that your machine has two or more processors, where the main processor is delegating another processor on the board to perform specific tasks. In older machines, the coprocessor was typically a math coprocessor. Newer processors have the math processor built into them. Modern coprocessors tend to be graphics processors or physics processors.

A modern machine has a CPU (central processing unit) anywhere from 1GHz to 3.6GHz, anywhere from 0.5GB to 8GB of RAM (either DDR, DDR2, or DDR3), and the typical HDD size is anywhere from 100GB to 1TB (1024GB). However the typical requirements of a GNU/Linux installation are as follows: 233MHz CPU, 64MB RAM, 2GB HDD. Recommended requirements would be more like 700MHz CPU, 256MB RAM, and 8GB HDD. Theoretically, one could have a GNU/Linux system running on a 486, with 4MB RAM, and a 500MB HDD. This, however, would be a single user system, and it would most likely lack a graphical user interface (the opposite of command line).

Spread Sheet Compilers
Calculators |
| |
V V
_________________________________
| Bash | Mail and
| ________________________ |<- Message
Inventory | | Linux kernel | | Facilities
Control --->| | ________________ | |
Systems | | | | | |<- Interpreters
| | | Hardware | | |
Formatters ->| | | | | |<- DBMS
| | | | | |
Calendar | | |______________| | | Word
Systems ---->| | | |<- Processors
| |______________________| |
Editors ---->| |<- FTP
|_______________________________|

The small and rather simple diagram above will give you a conceptual idea of how Unix and Unix-like systems (such as linux) operate. The operating system is a set of programs that act as a link between the computer and the user. The programs that allocate the system resources and coordinate all the details of the computer's internals is called the operating system or kernel. Users communicate with the kernel through a program known as the shell. The shell is a command line interpreter; it translates commands entered by the user and converts them into a language that is understood by the kernel. Some systems use a GUI (graphical user interface) instead of a command line interpreter but the idea is much the same. The design principles of Unix and Unix-like systems are as follows:

1. Make each program do one thing well. These simple
programs would be called "tools."
2. Expect the output of every program to be the input to
another program.
3. Don't stop building new "tools" to do a job. The
library of tools should keep increasing.

Unix systems around the world are united by a few common things, but in the past ten years many systems have started to become very non-Unix while their owners/maintainers still call them Unix.

1. THE KERNEL
The kernel is the heart of the operating system. It schedules tasks and manages data storage. The user rarely interfaces with the kernel directly. This is the memory resident portion of the operating system.

2. THE SHELL
The shell is the utility that processes your requests. When you type in a command at your terminal, the shell interprets the command and calls the program that you want. The shell will support multiple users, multiple tasks, and multiple interfaces to itself. The shell uses standard syntax for all commands. There are many popular shells currently available, the BourneShell (standard System V UNIX), the CShell (BSD UNIX), BASH (the GNU standard shell), ZSH (GoboLinux default shell), etc... Because separate users can use different shells at the same time, the system can appear different to different users. There is another shell known as the KornShell (named after its designer), which is popular with programmers. This ability to provide a customized user interface is one of the most powerful features of BSD / UNIX / Linux / GNU / Darwin / Solaris / AIX.

3. COMMAND & UTILITIES
Separate utilities can be easily combined to customize function and output. They are flexible, adaptable, portable, and modular. They use pipes and filters. There are over 200 standard commands plus numerous others provided through 3rd party software.

4. EVERYTHING IS A FILE
The directory system supports a multilevel hierarchy. Files and directories have access protection. Files and directories are accessed through pathnames. Files support multiple name links. Removable filesystems are also supported. But, it is important to note that in Linux/Unix everything is a file. Devices, directories, everything! To the OS it makes no difference what something's function is. This is important for output concatanation.

5. DIRECTORY HIERARCHY

/
(root)

bin dev etc lib tmp usr
home root sbin var opt boot mnt
media proc


Bin is for standard binaries. Dev holds device files. Etc Generally holds systemwide config files, and the init system files. Tmp is for temporary files. Lib holds system libraries. Usr holds user space programs. Home is the directory containing individual user directories (think My Documents). Root is the system administrator's home directory. Sbin contains system binaries and is not present in every system. Var holds data that is read/written frequently (variable data). Opt holds optional packages, this
is another directory that is not always present. Boot is almost always present (though not always {example of it not being included is on some LiveCDs}); /boot normally contains the system's boot loader and boot loader configuration files. Mnt is used for storage devices that are currently usable (aka mounted storage devices). Media is used for removable storage devices (this is another directory that isn't always present). There are exceptions to this hierarchy, however this is the most common way for Unix and Unix-like systems to structure their data. GoboLinux is a good example of an exception to the rules. GoboLinux is structured as follows:

/
(root)

Depot Files Programs Users Mount System

In this scheme Depot is used for Miscellaneous trappings (similar to opt). Files essentially functions as /etc. Programs is what truly sets this scheme apart from the rest. Each program gets its own directory within /Programs. The practical upshot of this is that installation and removal of the program becomes far easier and a system administrator can have multiple versions of the same program installed and running simultaneously. /Users functions as both /root and /home. Mount functions as both /media and /mnt. /System functions as /dev /tmp /var /proc /etc and /boot. For example your hardrive might be in /System/Kernel/Devices/hda1.

A file name can be upto 255 characters long and can use any character typable (except for * $ / \ & > < | @). The entire OS is case sensitive and as such so are filenames. 6. COMMAND BASICS First you have the command, then the options, and occasionally an arguement. A good example is ls, which lists the contents of a directory. ls -RabslFSH /home/bob Breaking this line down we see that the command was ls, the options were RabslFSH, and the arguement was /home/bob. This particular string would list all files that are in the directory and the files in all subdirectories (R), it would also list hidden files (a), it would use formatting characters for spaces and any other non-printable character (b), it would list out permissions (l), it would give you the file type (F), it would give you the size (s), it would sort by size (S), and it would give you the sizes in human-readable format (H). The /home/bob says that we are going to execute this command to affect the directory /home/bob (user bob's home directory {known in the Win$hit world as 'My Documents'}). This command will usually yield a ton of resultant output, especially if executed in the / directory. So, you may want to have this directory print to file instead of the screen. To do this we will execute a short series of commands.

$ touch directory_printout.txt
$ echo "DIRECTORY HIERARCHY PRINTOUT" > directory_printout.txt
$ ls -abslRSFH / > directory_printout.txt
$ cat directory_printout.txt | less

In this example, we are doing a few things that may seem cryptic at first. There is also a far easier way to do this, but I want to teach you a few concepts first. With 'touch directory_printout.txt' we are simply creating a text document, and the txt suffix is merely for human convenience. With 'echo "DIRECTORY HIERARCHY PRINTOUT" > directory_printout.txt' we are placing a header in the text document. Then we have our same old ls command complete with options and arguement, and we are sending the output to our new text document where it will be saved. The next command 'cat' is going to print the results on screen, and the "|" will pipe that output into the program
"less" which is a pager. Less/more/most will all take the output of a program and show only one page at a time, you can then press space bar to see the next page; if you press "q" the program quits. You can also have multiple elements in the arguement. For example:
$ diff memo.txt memo2.txt

In this example the two files are being compared for differences. Also, options are not always a single character. Occasionally, you'll come across a program whose options are whole words. For example:
$ find . -atime +7 -print

In this example we are looking for a file in the current working directory, that has been accessed within the past 7 days, and we are telling the program to list the directory name as well as the file name. If at any time you make a mistake while typing a command, you have several options, because nothing is done until you hit the enter key. Most of the time you can press backspace or delete in order to remove something from the line. If you are using a shell that does not support these keys (or a really strange key map), you can press the # key. This has the effect of a backspace. If you want to delete three characters back just press ###. Simple right? If you wish to delete an entire line you can press the @ key. Same idea.

If you need to end a program for some reason, you can almost always press CTL+c. This does not always work in a graphical environment, but it almost always works in the command line interface.

7. GETTING MORE INFORMATION ABOUT A PROGRAM

Generaly speaking, a user can get a little more information about a program's usage options just by typing '-h' '--h' '-help' or '--help' as an option for the command in question. However, occasionally you need more than just usage info. In such a case you can type $man CommandInQuestion

In the even that the command does not have a man page, or you do not have man installed, you can always use Google. Many people have many resources available online in some form or another.

8. PERMISSIONS

In Linux/Unix a file can be given permissions. These permissions can limit which users have the ability to read, write, and/or execute a file. So, if you were at the CLI and you typed "ls -l" you would get a read out on the permissions of the files. It would go as so: "rwx-rwx-rwx." The first grouping is for the owner, the second is for the group to which the user belongs, and the third is for others. R = read permission, W = write permission, X = execute permission.

In general, one should never login as "root," (the super user account). This is because while you are logged in as root everything can be modified (because you have permissions to access everything when you are root). So, if someone were to gain access to your system he/she could potentially harm you computer. Even if you do log in as root, you should NEVER EVER use X11 or any type of chat, or other network tool while root. These programs give outsiders easy tickets to your machine/data. If you need to do something that requires higher permissions consider using sudo, su, gksu, or kdesu (these will grant temporary permissions).

9. COMMANDS IN BRIEF

$ cd
change directory
$ ls
list directory contents
$ mv
move and/or rename
$ cp
copy
$ cat
show text contents
$ locate
locate a file or directory
$ find
find something
$ chroot
change root filesystem
$ exit
logout
$ modprobe
load a module
$ ps
show running processes
$ lsof
list open files
$ lsmod
list loaded modules
$ grep
find data within a file
$ ./foo
run foo
$ sh foo
run foo
$ startx
enter graphical environment
$ kill -9 1024
kill pid 1024
$ killall foo
kill all processes named foo
$ ifconfig -a
show ip configuration for all devices
$ iwconfig
show ip configuration for all wireless devices
$ dhcpcd eth0
run dhcp on eth0
$ dhclient eth0
run dhcp on eth0
$ ifup eth0
bring eth0 interface on
$ ifconfig eth0 up
bring eth0 interface on
$ wget
retrieve file via http
$ ftp
retrieve file via ftp
$ useradd
add a user (adduser)
$ groupadd
add a group (addgroup)
$ chmod
change permissions of a file
$ usermod
modify a user
$ passwd
change a password

Sunday, July 19, 2009

Your Linux Distribution

A lot of people are unsatisfied with the current distribution offerings that litter that software landscape. It can be frustrating to boot up a distro, only to realize you hate it and wasted a darn good CD to test it out. For this reason, I began trying to craft my own system. The only thing I can say to summarize is that you have a couple of ways to start this journey:
1. Use a currently available distribution and customize it to your needs
2. Start completely from scratch, and download each source
3. Mix 1 and 2
4. Use 2 but also borrow some scripts from people

If you want to go at things completely from source, I recommend reading Linux From Scratch. It's a good tutorial on compiling a Linux system. However, it is fairly involved. It's not a task for the light hearted. I have gone through it, but for what it has end up with, the root filesystem size is a bit too large for me. If you want to go the route of a full fledged desktop OS, you can use it in conjunction with BLFS (Beyond Linux From Scratch). Note that LFS does not offer package management. If you want package management you would have to do it yourself. I recommend Slackware's pkgtools for those who are too lazy to make their own package manager. If you want a kernel, you can hit up linux-live.org and use the kernel from Slax. It's already configured for a wide array of machine's and ought to make things easier for you. On PPC architectures, the Slackintosh kernel is very nicely put together.

Using current distributions is by far the easiest approach. Both Ubuntu and Fedora have programs to aid you in that endeavor. Debian Live also offers a quick and easy solution. Slackware is by far the best though. Slackware's package manager has an easy to use makepkg script, and the installer is wickedly easy to customize to meet your needs. Alright, so we start with the ISO file you download of the Slackware DVD. Open a terminal, and loop mount the ISO, then list the files

# mount -o loop SlackwareDVD.iso /mnt/cdrom -t \
auto && ls /mnt/cdrom

You ought to get something similar to the following:

ANNOUNCE.12_2
BOOTING.TXT
CHANGES_AND_HINTS.TXT
CHECKSUMS.md5
CHECKSUMS.md5.asc
COPYING
COPYING3
COPYRIGHT.TXT
CRYPTO_NOTICE.TXT
ChangeLog.txt
FAQ.TXT
FILELIST.TXT
GPG-KEY
PACKAGES.TXT
README.TXT
README.initrd
README_CRYPT.TXT
README_LVM.TXT
README_RAID.TXT
RELEASE_NOTES
SPEAKUP_DOCS.TXT
SPEAK_INSTALL.TXT
Slackware-HOWTO
UPGRADE.TXT
extra/
isolinux/
kernels/
pasture/
patches/
slackbook/
slackware/
source/
testing/
usb-and-pxe-installers/


Go ahead and make a build directory for yourself, as well as a package creation directory, then move into your build directory.
Copy the contents of the DVD into your build directory.

$ mkdir ~/build
$ mkdir ~/packages
$ cd ~/build
$ cp -R /mnt/cdrom/* ~/build


Now comes the "fun" part. If you move into the slackware/ directory you are greeted by an ugly assortment of files and packages. The installer in isolinux/ reads these package directories and files to install packages in the target HDD. For it to do this, each package series' directory contains a few unique files: install, install.end, maketag, maketag.ez, and tagfile. You will not need to touch install or install.end, but do not delete either file. maketag and maketag.ez are identical. tagfile is different from the other two, but is mostly just a list. Every package you add or remove has to be listed in all three of those files. You can look at each one and get a good sense of how to edit them. You can remove an entire series of packages if you wish, but you will need to make an adjustment to the installer if you do. For example, I prefer to use CLI only systems. I always remove X. Move into isolinux/ and look for 'setpkg'. You need to remove any package series from this file if you removed them from slackware/ --ie if you deleted x/ you will want to remove line 32 and line 33 (X and XAP):

"X" "X Window System" on "This series contains X, the window system (or GUI) used by Linux." \
"XAP" "X Applications" on "The XAP series is a collection of applications for X." \


So, if all you are doing is changing the package selection, Slackware can be a good basis. More can be done though. You can change the installer by extracting the initrd.img file in isolinux/

$ cd ~
$ mkdir extracted
$ cd extracted
$ zcat ../build/isolinux/initrd.img | cpio -div


You will find the installation scripts in extracted/usr/lib/setup. I am not going to detail each file, as that is beyond the scope of my post, but if you search for lines that start with --dialog, you will have a good idea of what is going on. Reassembling your initrd is easy:

$ cd ~/extracted
$ find . | cpio -o -H newc | gzip -9fv > ../build/isolinux/initrd.img


Making packages is relatively easy but it changes for each piece of software you wish to add. For the majority you can download the source, extract it, and then cd into the new directory, compile and then run makepkg. Example:

$ wget http://www.gnu.org/some/piece/of/software.tar.gz
$ tar -xzf software.tar.gz
$ cd software/
$ ./configure --prefix=/home/username/packages
$ make && make install
$ cd ~/packages
$ makepkg software.tgz

I would now move software.tgz into the slackware/ folder in ~/build and add it to the maketag files and the tagfile.

To assemble the ISO image:

$ cd ~/build
$ mkisofs -o ~/myDistro.iso -R -J -V "myDistro" \
-hide-rr-moved -v -d -N -no-emul-boot \
-boot-load-size 32 -boot-info-table \
-sort isolinux/iso.sort -b isolinux/isolinux.bin \
-c isolinux/isolinux.boot -A "myDistro" .

Do NOT forget the trailing period after the last "myDistro"

Yet another ways is to use some scripts for automation of certain software package's compilation and do things from scratch otherwise. You can still use the Slackware install disc, but you would replace the setup scripts in the initrd with your own scripts (for your own package manager unless you want to use Slackware's). You could also use linux-live scripts to generate a liveCD and you could even use the linux-live kernel. BusyBox + buildroot can help you build your root environment, although "linux from scratch embedded in < 40mins" is another good tutorial. Crosstool can give you a prebuilt cross compiler to help you compile things, and you can use the TinyX script from the freedesktop.org website to build X11. From there, the sky is the limit. You can write your own software to go in your system, or you can compile other's software.

If you were to use buildroot + TinyX script + mutt + links2 + rxvt + lwm/evilwm and then roll that up with Linux-Live you would have a system you could install to a thumb drive and take with you where ever you wanted to go.