Sunday, 20 October 2013

Centos 6/RHEL install Hadoop Single Node Server

Installation of Hadoop on your Centos 6/RHEL box is now a lot simpler since rpm versions have been made available but you nonetheless need to have installed the JDK prior to doing so.

Change the JAVA_HOME path to /usr/java/default and you can install Hadoop via yum from Epel repo.

$ sudo yum -y install hadoop

If you have any problems with yum you can also use the Apache mirror service, download your preference and install it with

$ sudo rpm -Uvh <rpm_package_name>

Once installed as a package, set it up in two steps.

First set up configuration and start Hadoop processes

$ sudo /usr/sbin/hadoop-setup-single-node.sh 

You will be asked a series of questions to which you should answer yes and you should get output similar to below.

Welcome to Hadoop single node setup wizard

Would you like to use default single node configuration? (y/n) y
Would you like to format name node? (y/n) y
Would you like to setup default directory structure? (y/n) y
Would you like to start up Hadoop? (y/n) y
Would you like to start up Hadoop on reboot? (y/n) y

Review your choices:

Setup single node configuration    : y
Format namenode                    : y
Setup default file system structure: y
Start up Hadoop                    : y
Start up Hadoop on reboot          : y

Proceed with setup? (y/n) y


and lots more besides, after which the setup will be complete and Hadoop should be started.

Now create a user account on the HDFS

$ sudo /usr/sbin/hadoop-create-user.sh -u $USER 

So setting up a single node Hadoop cluster is now much easier.









Centos 6/RHEL install Baobab Disk Analysis

Baobab is a useful graphical tool for checking the amount of disk space used on your system. It is available in the Epel repo as part of the gnome-utils package which contains a set of small desk accessory utility applications for GNOME, such as a dictionary, a disk usage analyzer, a screenshot tool and others.

Make sure that you have it installed with one of the following commands.

$ sudo yum -y install baobab

$ sudo yum -y install gnome-utils

Now you can use it with a couple of simple commands.

$ baobab /home/anton

Baobab Centos 6
$ baobab

Baobab
Baobab places disk usage analysis in two panels.

Per-directory disk utilization stats are reported in numbers in the left panel whereas the overall disk utilization is visualized in the right panel.


In the ring chart right panel visualisation, multiple concentric circles show the directories from the directory tree, while the center of the circles corresponds to the root of the directory tree. 

The smaller a circle is, the higher a corresponding directory ranks in the directory hierarchy. Hovering a mouse over any particular directory will show the names of its sub-directories on the immediate outer circle. The relative sizes of the sub-directories can be assessed by comparing their sizes in a ring-shaped area.


Baobab also uses treemap charts in which overlapping rectangles are implemented to visualize the hierarchical structure of directories, where the outermost rectangle corresponds to the root of the directory tree, and the inner rectangles to sub-directories. The size of each rectangle indicates the size of a corresponding sub-directory. Again, you can hover a mouse to check the name of a sub-directory.


Baobab can analyze local folders as well as remote folders. To examine remote folders, click on “Scan a Remote Folder” button at the far right. Baobap supports SSH, FTP, WebDAV and Windows Share.

Sunday, 13 October 2013

Centos 6/RHEL install Nagios


Nagios is a powerful monitoring system that enables you to identify and resolve infrastructure problems before they affect critical processes.

Designed with scalability and flexibility in mind, Nagios gives you the peace of mind that comes from knowing your business processes won't be affected by unknown outages.

Nagios is a powerful tool that provides you with instant awareness of your organization's mission-critical IT infrastructure. It allows you to detect and repair problems and mitigate future issues before they affect end-users and customers.

By using Nagios, you can:

  • Plan for infrastructure upgrades before outdated systems cause failures
  • Respond to issues at the first sign of a problem
  • Automatically fix problems when they are detected
  • Coordinate technical team responses
  • Ensure your organization's SLAs are being met
  • Ensure IT infrastructure outages have a minimal effect on your organization's bottom line
  • Monitor your entire infrastructure and business processes 

    To use Nagios, first make the necessary alterations to your network configuration file. 

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
Change
ONBOOT=no to ONBOOT=yes
Add the following to the configuration file, fill the X’s to your IP configuration:

IPADDR=X.X.X.X
GATEWAY=X.X.X.X
NETMASK=X.X.X.X
NETWORK=X.X.X.X
 
Save and close the file, then restart the network service.


# service network restart
Now install the nagios related packages

# yum install httpd php gcc glibc glibc-common gd gd-devel openssl-devel wget perl make
Create the nagios user and set the password

# /usr/sbin/useradd -m nagios
 
# passwd nagios

Create the nagios group, nagcmd, which will allow external commands to be submitted through the web interface. 
Then add the nagios user and apache user to the group
 
# /usr/sbin/groupadd nagcmd

# /usr/sbin/usermod -a -G nagcmd nagios

# /usr/sbin/usermod -a -G nagcmd apache

Create a temp directory to place the nagios downloads

# mkdir ~/downloads cd ~/downloads

Download nagios


# wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.5.0.tar.gz  
Extract the nagios tar ball

 
# cd nagios
  
Run the nagios script passing the nagcmd group


# ./configure --with-command-group=nagcmd
General Options: ------------------------- Nagios executable: nagios Nagios user/group: nagios,nagios Command user/group: nagios,nagcmd Embedded Perl: no Event Broker: yes Install ${prefix}: /usr/local/nagios Lock file: ${prefix}/var/nagios.lock Check result directory: ${prefix}/var/spool/checkresults Init directory: /etc/rc.d/init.d Apache conf.d directory: /etc/httpd/conf.d Mail program: /bin/mail Host OS: linux-gnu Web Interface Options: ------------------------ HTML URL: http://localhost/nagios/ CGI URL: http://localhost/nagios/cgi-bin/

Compile the nagios source code

# make all

Install the binaries, init script, sample configs, and set permissions on external command directory


# make install

# make install-init

# make install-config

# make install-commandmode
 
The sample config files have been installed in the /usr/local/nagios/etc directory. 
Edit the /usr/local/nagios/etc/objects/contacts.cfg file and change the email address associated with the nagiosadmin contact definition to the address you’d like to use for receiving alerts.

# vi /usr/local/nagios/etc/objects/contacts.cfg
Configure the web interface

# make install-webconf
Create the nagiosadmin account for logging into the web interface


# htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
Restart Httpd


# service httpd restart
Download the plugins 
# tar xzf nagios-plugin (tab)

# cd nagios-plugins (tab)
 
Compile and install the plugins


# ./configure --with-nagios-user=nagios –with-nagios-group=nagios

# make

# make install
 
Add nagios to the system services and have it start when system boots


# chkconfig --add nagios

# chkconfig nagios on
 
Verify the nagios configuration files


# /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

# service nagios start
Find out if you have SELinux in Enforced mode

getenforce

Run the following commands to run the CGIs under SELinux enforcing/targeted mode

chcon -R -t httpd_sys_content_t /usr/local/nagios/sbin/
chcon -R -t httpd_sys_content_t /usr/local/nagios/share/
 
Log into the nagios web interface

http://localhost/nagios
Username will be nagiosadmin unless you specified different, plus the password you have set. 
You can also add numerous extras from Nagios Exchange.


If you try to access the web interface remotely and the page doesn’t load, check to see if you have IPTABLES enabled

service iptables status
If you don’t see entries for port 80 and/or 443, then it is being denied.

Allow http and https services through IPTABLES:

vi /etc/sysconfig/iptables
Add the following entries:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

Sunday, 6 October 2013

Centos 6/RHEL popular commands


One of the greatest attributes of the Minimal Fluxbox Desktop is the key shortcuts to the most used programs and these are certain to include the likes of Thunar and Rxvt terminal emulator or similar. You can use the F1 through 12 keys amongst others to pop up the programs and I assign F1 and F2 to Thunar and Rxvt respectively. 

Notwithstanding the fact that the Minimal Desktop can be made headless by simply hitting Ctrl+Alt+F4, there are also a formidable array of commands at your disposal in order to perform tasks from the console or terminal emulator. Below are some of the most used ones.


The following list describes some of the most useful and popular Linux commands. Consult the man page for each command to learn about additional arguments and details of operation.

# adduser userid

Creates a new userid, prompting for necessary information

# apropos keyword
Searches the manual pages for occurrences of the keyword and prints short descriptions from the beginning of matching manual pages.
 
# at time
 
# at -t file time
 
Executes commands entered via stdin (or, by using the alternative form, the specified file) at the specified time. The time can be specified in a variety of ways; for example, in hour and minute format or in hour, minute, month, day, and year format.
 
# cal month year 

Prints a calender for the specified month of the specified year

Centos 6 calender function
# cat files
 
Prints out the contents of the specified files
 
# cd
 
# cd directory
 
Changes the current working directory to the user's home directory or the specified directory.
 
# chgrp group files
 
# chgrp -R group files
 
Changes the group of the specified files to the specified group. The alternative form of the command operates recursively, changing the group of subdirectories and files beneath a specified directory. The group must be named in the /etc/groups file, maintained by the newgroup command
 
# chmod mode files
 
# chmod -R mode files  
 
Changes the access mode of the specified files to the specified mode. The alternative form of the command operates recursively, changing the mode of subdirectories and files beneath a specified directory.
 
# chown userid files
 
# chown -R userid files 
 
Changes the owner of the specified files to the specified userid. The alternative form of the command operates recursively, changing the owner of subdirectories and files beneath a specified directory.
 
# clear
 
Clears the terminal screen
 
# cp file1 file2

# cp files directory
 
# cp -R files directory
 
Copies a file to another file or directory, or copies a subdirectory and all its files to another directory.
 
# date
 
# date date
 
Displays the current date and time or changes the system date and time to the specified value.
 
# df
 
Prints the amount of free disk space on each mounted filesystem
 
# diff file1 file2 

# Compares the two files, reporting all discrepancies. Similar to the diff command though the output format differs.
 
# dmesg
 
Prints the messages resulting from the most recent system boot
 
# du
 
# du directories 

Prints the amount of disk space used by the current directory (or the specified directories) and subdirectories.

# echo string

# echo -n string

Prints the specified text on the standard output stream. The -n option causes omission of the trailing newline character.
 
# fdformat device 

Formats the media inserted in the specified floppy disk drive. The command performs a low-level format only; it does not create a filesystem. To create a filesystem, issue the mkfs command after formatting the media.

# fdisk device

Edits the partition table of the specified hard disk.

# fg

# fg jobs

Brings the current job (or the specified jobs) to the foreground.

# file files

Determines and prints a description of the type of each specified file.
 
# find

# find path -name pattern -print

Search for files in a directory hierarchy. Searches the specified path for files with names matching the specified pattern (usually enclosed in single quotes) and prints their names.
 
# finger users 
 
Prints a description of the specified users (install with yum).
 
Centos 6 Finger command

# ftp hostname

Opens a FTP connection to the specified host, allowing files to be transfered. The FTP program provides subcommands for accomplishing file transfers.

# grep pattern files
 
# grep -i pattern files
 
# grep -n pattern files
 
# grep -v pattern files 
 
Search the specified files for text matching the specified pattern (usually enclosed in single quotes) and print matching lines. The -i option specifies that matching is performed without regard to case. The -n option specifies that each line of output is preceded by the file name and line number. The -v option reverses the matching, causing non-matched lines to be printed.

# gzip files

# gunzip files

Compress (or expand) the specified files. Generally, a compressed file has the same name as the original file, followed by .gz.
 
# head files
 
Prints the first several lines of each specified file.
 
# hostname
 
# hostname name
 
Displays (or sets) the name of the host.

# info

Launches the GNU Texinfo help system.

# init run_level (eg '3')

Changes the system run level to the specified value

# insmod module

Dynamically loads the specified module

# jobs

Displays all background jobs

# ispell files

Checks the spelling of the contents of the specified files

# kill process_ids

# kill -l

Kills the specified processes, sends the specified processes the specified signal (as a number or a name), or prints a list of available
 
# killall program
 
Kills all the processes that are instances of the specified program or sends the specified signal to all processes that are instances of the specified program.

# ispell files

Checks the spelling of the contents of the specified files

# ln old new

# ln -s old new

Creates a hard (or soft) link associating a new name with an existing file or directory.

# locate pattern

Locates files with names containing the specified pattern and uses the database maintained by the updatedb command.

# lpq

Prints the entries of the print queue

# lpr files

Prints the specified files

# lprm job

Cancels printing of the specified print queue entries. Use lpq to determine the contents of the print queue.

# ls

# ls files

# ls -a files

# ls -l files

# ls -lr files

Lists (non-hidden) files in the current directory or the specified files or directories. The -a option lists hidden files as well has non-hidden files. The -l option causes the list to include descriptive information, such as file size and modification date. The -R option recursively lists the subdirectories of the specified directories.
 
# mail
 
Launches a simple mail client that permits sending and recieving email messages. 

# man title

# man section title

Prints the specified man page.

# mkdir directories

# mkdir -p directories

Creates the specified directories. The -p option causes creation of any parent directories needed to create a specified directory.
 
# mkfs -t type device 
 
Creates a file system of the specified type on the specified device.

# mkswap device
 
Creates a Linux swap space on the specified hard disk partition.

# more file
 
Lets the user peruse a file too large to be displayed as a single screen (page) of output. The more command provides many subcommands that let the user navigate the file. For example, the Space key moves forward one page, the b key moves back one page, and the q key exits the program.

# mount

# mount device directory

# mount -o option -t device directory
 
Prints the mounted devices or mounts the specified device at the specified mount point. (usually a sub-directory of /mnt).
The mount command consults /etc/fstab to determine statndard options associated with a device. The command generally requires root privileges. The -o option allows specification of a variety of options. For example, ro for read-only access. The -t option allows specification of the filesystem type (eg ext2, msdos etc)
 
# mv paths target
 
Moves the specified files or directories to the specified target.
 
# newgroup 

Creates the specified group

# passwd

# passwd user
 
Changes the current user's password, or that of the specified user (reaquires root privileges). The command prompts for the new password.
 
# ping host
 
Sends an echo request via TCP/IP to the specified host. A response confirms that the host is operational.
 
# pr files
 
Formats the specified files for printing, by inserting page breaks and so on. The command provides many arguments and functions.
 
# ps
 
# ps -Aux
 
Display the processes associated with the current userid or displays a description of each process.
 
# pwd
 
Prints the absolute path corresponding to the current working directory. 

# reboot

Reboots the system.

# reset

Clears the terminal screen and resets the terminal status.

# rm files

# rm -i files

# rm -f files

# rm -if files

# rm -rf files

Deletes the specified files or (when the -r option is specified) recursively deletes all subdirectories of the specified files and directories. The -i option causes the command to prompt for confirmation; the -f option suppresses confirmation. Because deleted files cannot generally be recovered, the -f option should be used only with extreme care, particularly when used by the root user.
 
# rmdir directories 

# rmdir -p directories

Deletes the specified empty directories or (when the -p option is specified) the empty directories along the specified path.
 
# shutdown minutes
 
# shutdown -r minutes 

Shuts down the system after the specified number of minutes elapses (requires root privileges). The -r option causes the system to be rebooted once it has shut down.
 
# sleep time
 
Causes the command interpreter to pause for the specified number of seconds
 
# sort files
 
Sorts the specified files. The command has many useful arguments.
 
# split file

Splits a file into several smaller files. The command has many arguments.

# su

# su user

# su -

# su - user

Changes the current userid to root or to the specified userid
The - option establishes a default environment for the new userid.

# swapon device

Enable use of the specified device for swapping

# swapoff device

Enables use of the specified device for swapping

# sync

Completes all pending input/output operations

# tail file

# tail - n file

# tail -f file
 
Prints the last several lines of the specified files. The -n option specifies the number of lines to be printed. The -f option causes the command to continuously print additional lines as they are written to the file.
 
# talk user
 
Launches a program that allows a chat-like dialog with the specified user
 
# tar cvf tar_file files
 
# tar zcvf tar_file files 

Creates a tar file with the specified name, conCreates the specified directories. The -p option causes creation of any parent directories needed to create a specified directory.taining the specified files and their subdirectories. The z option specified that the tar file will be compressed.
 
# tar xvf tar_file
 
# tar zxvf tar_file 

Extracts the contents of the specified tar file. The z option specified that the tar file has been compressed.

# telnet host

Opens a login session on a specified host

# top

Prints a display of system processes that is continually updated until the user presses the q key.

# traceroute host
 
Uses echo requests to determine and print a network path to the host
 
# umount device
 
Unmounts the specified filesystem
 
# uptime
 
Prints the system uptime
 
# w

Prints the current system users

# wall

Prints a message to each user except those who've disabled message reception
 
# traceroute host
 
Uses echo requests to determine and print a network path to the host. 

# wc files

Prints the number of characters, words and lines in the specified files.




Thursday, 3 October 2013

Centos 6/RHEL using man pages

In Centos 6/RHEL and other Linux distros, command line programs come with their own documentation called manual pages or man pages. These are generally written by the developer of the corresponding program and are divided into number of sections.

Below is the list of available man sections. Every section has a unique number and contains only a specific type of man pages.
  • 1 – Executable programs or commands
  • 2 – System calls ( functions provided by the kernel )
  • 3 – Library calls ( functions provided by the library )
  • 4 – Special files
  • 5 – File formats and conventions ( configuration files )
  • 6 – Games
  • 7 – Miscellaneous
  • 8 – System administration commands
Syntax:

$ man <topicname>

View the Man Page of a command

To read the man page of an Linux command, pass the command name as the argument to the man. The following will display the man page for passwd command.

$ man passwd
PASSWD(1) User utilities  PASSWD(1)
NAME
    passwd - change user password

Some topics may even have man pages in more than one section. In such a case, man command will display the page which has lower section number.
In this example, the passwd command has manual pages in multiple sections. But, by default, it displays the man page from the section 1.
The “PASSWD(1)” shown in the 1st line of the man command output indicates that it is displaying the man page from section 1. The man page output displays the command name, syntax of the commands, description of what the command does, options provided by the command, etc…

View Man Page from a Specific Section

To read the man page from a particular section, provide the section number as follows. The passwd command has man page in both section 1 and section 5. By default, if you don’t specify the section number, it will display man page from section 1.
To display man page from section 5, specify the section number as shown below.

$ man 5 passwd

Now it will display the manual page for /etc/passwd configuration file, since the section number 5 is for File Formats and Conversions.

List Available Man Sections for a Command

You can also list all the available sections on a particular topic using -aw option.

$ man -aw printf
/usr/share/man/man1/printf.1.gz

View All Man Pages for a Command – Display All Sections

To view all the man pages for a particular topic, use the “-a” option. You’ll see the lowest-number man page first. When you exit that page, and press “Enter” the next man page will appear.

$ man -a printf

Change the Default Pager used by Man Command

By default man command will use the $PAGER environment variable to identify which pager to use for showing output. User can change the pager in which they prefer to see the man page using ‘-P’ option.
The following command will display the man page using more command pager.

$ man -P more printf

Search Man Page against NAME Section

To search the man page against NAME section, use “-f” option as shown below.

$ man -f printf

printf (3) - formatted output conversion

This is equivalent to using whatis shell command.

The above command, searches the manual page names, and displays the description for the given topic if the manual page names, matches with the given topic. You can also pass multiple topics in the same command line.

Search Man Page against NAME and DESCRIPTION Section

To search the man page against NAME & DESCRIPTION section, use “-k” option. It is equivalent to using “apropos” shell command.

$ man -k printf

printf               (1)  - format and print data
printf [builtins]    (1)  - bash built-in commands, see bash(1)
printftest           (6)  - tests the vgagl gl_printf function
set_matchpathcon_printf [set_matchpathcon_flags] (3)  - set flags controlling the operation of matchpathcon or matchpathcon_index and configure the behaviour of validity checking and error displaying

The above command will search for the keyword “printf” as regular expression and display the man pages that match the keyword.