CurlFTPFS is a very good tool to mount remote FTP directory to the local file-system. CurlFtpFS is a filesystem for accessing FTP hosts based on FUSE and libcurl.

CurlFtpFS differentiates itself from other FTP filesystems because it features:

  • SSLv3 and TLSv1 support
  • connecting through tunneling HTTP proxies
  • automatically reconnection if the server times out
  • transform absolute symlinks to point back into the ftp file system

Use the following commands are used to install the package:

After you complete the script execution you need to setup/configure your FTP source.

1 – Create directory (this will be the mount point).

2 – Connect to the FTP Servers

  • this is how you would connect manually to the FTP Folder.

Note:-ftp uses unencrypted passwords so anyone can intercept your password.

3 – Mount FTP Folder permanently by adding it to the /etc/fstab file.

  • Create the .netrc conf file in your user home folder

  • Add this line to the /etc/fstab file 

  • Now to mount it run 

 

   Rsync is an open source utility that provides fast incremental file transfer. Rsync is typically used to synchronize files and directories between two different systems.

Base Syntax:

or

When on localhost only.

rsync

Why rsync is so good ?

  • Speed: First time, rsync replicates the whole content between the source and destination directories. Next time, rsync transfers only the changed blocks or bytes to the destination location, which makes the transfer really fast.
  • Security: rsync allows encryption of data using ssh protocol during transfer.
  • Less Bandwidth: rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.
  • Privileges: No special privileges are required to install and execute rsync.

 

Now that we know what is rsync and why should we use it let us go over some use examples.

 1 – How to Synchronize Files From Local to Remote

rsync allows you to synchronize files/directories between the local and remote system but both hosts must have rsync installed on them

Note: the -a option stand for “archive mode” , -z “compress file data during the transfer”

Note: while doing rsync with the remote server, you need to specify username and ip-address of the remote server. You should also specify the destination directory on the remote server.

2 – How to Synchronize Files From Local to Local

– all file from folder files will be copied to files2.

3 – How to Synchronize Files From Remote to Local

When you want to synchronize files from remote to local, specify remote path in source and local path in target as shown below.

4 – How to Synchronize Only One File

Note: you must specify the file name at the source

5 – How to Synchronize only the Directory Tree Structure (not the files)

You can sync just directories by excluding everything else.

6 – How to View the rsync Progress during Transfer with Rsync

For this you need to add the –-progress option

7 – How to Delete the Files Created at the Target with Rsync

If a file is not present at the source, but present at the target, you might want to delete the file at the target during rsync.

In that case, use –delete option. Rsync delete option deletes files that are not there in source directory.

8 – How to restrict Large Files transfers with Rsync

We can tell rsync not to transfer files that are greater than a specific size using rsync –max-size option.

We just put a limit on the max file size to be allowed to be transferred.

 

There more complex operations that rsync can be used for but i just covered some basic ones, fell free post any rsync examples that you find useful.

 

    New to Linux/Unix ? Or you worked with Linux/Unix but never got deep into Linux filesystem structures and the directories ? No problem in this article i will walk you thru what is their purpose, where they are located and how we can use them !

Bellow is image of the hierarchical layout having the / at the top

linux-file-system

 

 

I will start describing each of them starting with root aka

/ – > Root

  • Every single file and directory starts from the root directory.
  • Only root user has write privilege under this directory.
  • Please note that /root is root user’s home directory, which is not same as /.

/bin – > User Binaries

  • Contains binary executables.
  • Common linux commands you need to use in single-user modes are located under this directory.
  • Commands used by all the users of the system are located here.
  • For example: ps, ls, ping, grep, cp.

/dev – > Device Files

  • The /dev directory contains the special device files for all the devices. The device files are created during installation. All hardware files are present in /dev(Device ) folder. If we observe the /dev folder you can find files/folders related to different hardware’s present in the machine.

/home – Home Directories

  • Home directories for all users to store their personal files.
  • For example: /home/johndoe, /home/aodba

/mnt – Mount Directory

  • Temporary mount directory where sysadmins can mount filesystems.

/root – Root Home Directory

/tmp – Temporary Files

  • Directory that contains temporary files created by system and users.
  • Files under this directory are deleted when system is rebooted.

/var – Variable Files

  • var stands for variable files.
  • Content of the files that are expected to grow can be found under this directory.
  • This includes — system log files (/var/log); packages and database files (/var/lib); emails (/var/mail); print queues (/var/spool); lock files (/var/lock); temp files needed across reboots (/var/tmp);

/boot – Boot Loader Files

  • Contains boot loader related files.
  • Kernel initrd, vmlinux, grub files are located under /boot

/etc – Configuration Files

  • Contains configuration files required by all programs.
  • This also contains startup and shutdown shell scripts used to start/stop individual programs.
  • Contains device files.
  • These include terminal devices, usb, or any device attached to the system.

/lib – System Libraries

  • Contains library files that supports the binaries located under /bin and /sbin
  • Library filenames are either ld* or lib*.so.*

/proc – Process Information

  • Contains information about system process.
  • This is a pseudo filesystem contains information about running process. For example: /proc/{pid} directory contains information about the process with that particular pid.
  • This is a virtual filesystem with text information about system resources. For example: /proc/uptime

/sbin – System Binaries

  • Just like /bin, /sbin also contains binary executables.
  • But, the linux commands located under this directory are used typically by system aministrator, for system maintenance purpose.
  • For example: iptables, reboot, fdisk, ifconfig, swapon

/usr – User Programs

  • Contains binaries, libraries, documentation, and source-code for second level programs.
  • /usr/bin contains binary files for user programs. If you can’t find a user binary under /bin, look under /usr/bin. For example: at, awk, cc, less, scp
  • /usr/sbin contains binary files for system administrators. If you can’t find a system binary under /sbin, look under /usr/sbin. For example: atd, cron, sshd, useradd, userdel
  • /usr/lib contains libraries for /usr/bin and /usr/sbin
  • /usr/local contains users programs that you install from source. For example, when you install apache from source, it goes under /usr/local/apache2

/opt – Optional add-on Applications

  • opt stands for optional.
  • Contains add-on applications from individual vendors.
  • add-on applications should be installed under either /opt/ or /opt/ sub-directory.

/srv – Service Data

  • srv stands for service.
  • Contains server specific services related data.
  • For example, /srv/cvs contains CVS related data.

 

 

In this article i will walk you thru the steps you need to format a new volume as ext4 format and make it available to your Linux box.

We have a volume called /dev/sdg and at the moment it needs formatting.

inode

To format the new volume to ext4 file system type we will use mkfs.ext4 utility as root user.

  • you need to provide him with the y falg to continue.

Note:

Now we can check that our volume thou is no mounted the file system type if ext4

  •  Great we can see that now we have the volume formatted with ext4 fs type.

Next we need to mount the new formatted volume to our Linux host

  • we are going to mount it on directory /vertica_storage/t1_data_03

We can see that our volume is mounted.

 

Add the new new mount point to /etc/fstab file.

The configuration file /etc/fstab contains the necessary information to automate the process of mounting partitions. In a nutshell, mounting is the process where a raw (physical) partition is prepared for access and assigned a location on the file system tree (or mount point).

Wanna know more about /etc/fstab file ? Read the article Fstab in detail.

 

When using Linux you will needed to change some options on your file systems. Getting acquainted with fstab can make the whole process a lot easier, and it’s much easier than you think.

The configuration file /etc/fstab contains the necessary information to automate the process of mounting partitions. In a nutshell, mounting is the process where a raw (physical) partition is prepared for access and assigned a location on the file system tree (or mount point).

The file FSTAB contains descriptive information about the various file systems. fstab is only read by programs, and not written; it is the duty of the system administrator to properly create and maintain this file. Each filesystem is described on a separate line; fields on each line are separated by tabs or spaces. Lines starting with ‘#’ are comments. The order of records in fstab is important because fsck, mount, and umount sequentially iterate through fstab doing their thing.

This is an example of a fstab content

fstab

There are total six columns in the fstab file separated by spaces or tabs. Each column holds different information about the device. For adding any new device add a fresh row. Each row stands for a partition or removable device in the system.

<file system>    

   Device driver’s path tells the system to mount the device with the mentioned identifier

<mount point>

     Shows the mount point specified for a device in the fstab file. The mount points actually is the directory where that particular device (mentioned in the first column) will be                                          mounted and through which we can view and modify the content of that partition.

<type>

  The third column in the file specifies the file system type of the device or partition.

<option>          

Holds values such as :

  •               strictatime(updates the access time of the files every time they are accessed)
  •               noatime(disables writing file access times to the drive every time you read a file)
  •               nodiratime(disables the writing of file access times only for directories while other files still get access times written)
  •               relatime(updates the access time only if the previous access time was earlier than the current modify or change time).
  •               lazytime(reduces writes to disk by maintaining changes to inode timestamps (access, modification and creation times) only in memory)

<dump>        

   Column is for backup option. This column contains either 0 or 1. Where “0” stands for “NO” and “1” stands for “YES”. The system checks it at the time of booting, if it’s “0”,                                         dump will ignore that filesystem but if its “1” then it will enable backup option. Backup is supported on only ext3 file system, hence only for ext3 file system it should be enabled                                     and for rest of the file systems it should be disabled.

<pass>  

  This column defines the order in which the system should scan the partitions on start up. The / partition is assigned top priority i.e. 1 and the rest of the partitions are assigned                                   second priority i.e. 2. If value is set to 0 means no scanning will be done at the time of startup. If same number is given to different partitions then the partitions are scanned                                           together with equal priority.

 

Now we can edit and manage your /etc/fstab file without having any problems. 

Note:

Always remember to make a backup in case something goes wrong, but have fun, and be sure to leave your experiences in the comments!

The ext(Extended File System) inspired by the traditional Unix File System (UFS) and was designed by Rémy Card, started the +2Gb file size limit era.

What is an Inode ?

The inode is a data structure used to represent a file system object, which can be one of various types including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object’s data. File system object attributes may include manipulation metadata, as well as owner and permission data.

inode

Ext2 (Second Extended File System)

  Ext2 does not have journaling feature and for this is recommended for flash drives, usb drives, as it doesn’t need to do the over head of journaling also you can use the the noatime mount option, for the same reason.

  The Maximum individual file size can be from 16 GB to 2 TB and overall ext2 file system size can be from 2 TB to 32 TB.

   The space in ext2 is split up into blocks. These blocks are grouped into block groups, analogous to cylinder groups in the Unix File System. There are typically thousands of blocks on a large file system. Data for any given file is typically contained within a single block group where possible. This is done to minimize the number of disk seeks when reading large amounts of contiguous data.

   Each block group contains a copy of the superblock and block group descriptor table, and all block groups contain a block bitmap, an inode bitmap, an inode table and finally the actual data blocks.

Example of ext2 inode structure:

 

 

ext2-inode

Ext3 (Third Extended File System)

ext3 allows journaling. Journaling has a dedicated area in the file system, where all the changes are tracked. When the system crashes, the possibility of file system corruption is less because of journaling.

The Maximum individual file size can be from 16 GB to 2 TB and overall ext2 file system size can be from 2 TB to 32 TB.(similar to ext2)

 Types of journaling available in ext3:

  • Journal        – Metadata and content are saved in the journal.
  • Ordered       – Only metadata is saved in the journal. Metadata are journaled only after writing the content to disk. This is the default.
  • Writeback   – Only metadata is saved in the journal. Metadata might be journaled either before or after the content is written to the disk.

You can convert a ext2 file system to ext3 file system directly (without backup/restore).

Drawbacks on ext3:

  • A directory can have at most 31998 subdirectories, because an inode can have at most 32000 links.
  • You cannot be fsck while the filesystem is mounted for writing.
  • ext3 does not support the recovery of deleted files.
  • ext3 does not have native support for snapshots(capture the state of the filesystem at a point in time).
  • ext3 does not do checksum when writing to the journal.

ext3

 

Ext4 (Fourth Extended File System)

The ext4 filesystem can support volumes with sizes up to 1 exbibyte (EiB) and files with sizes up to 16 tebibytes.

Extents are added in ext4. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation. A single extent in ext4 can map up to 128 MiB of contiguous space with a 4 KiB block size.

You can mount ext3 or ext2 file systems as ext4(Backward compatibility).

Ext4 can pre-allocate on-disk space for a file(applications like media streaming and databases make great use of this).

Journal checksumming is available now and improves journal reliability.

Improved  file system checking due to skipping unallocated block groups and sections.

Journal can be disabled.

 

In CentOS 7 the process called Firewalld .

 FirewallD is front-end controller for iptables used to implement persistent network traffic rules. It provides command line and graphical interfaces and is available in the repositories of most Linux distributions.

If you want to know more about it follow the link above.

 

 

How we Disable Firewalld:
  • login as root user and run the command bellow:

How to Stop the Firewalld:
  • login as root user and run the command bellow:

Check the Status of Firewalld
  • login as root user and run the command bellow:

Quite simple and easy but you need to have other ways of protecting after disabling the firewall.

     I am going to walk you guys thru how we can install additional standard open source packages by using yum command.

For those curious:

     EPEL (Extra Packages for Enterprise Linux) is open source and free community based repository project from Fedora team which provides 100% high quality add-on software packages for Linux distribution including RHEL (Red Hat Enterprise Linux), CentOS, and Scientific Linux.

More info at EPEL.

EPEL repository is great because it has lots of open source packages and most important is free !

So how can we use EPEL on our Linux machines?

We need to download the repository file using wget and install it using rpm on your system to enable the EPEL repository.

RHEL and CentOS versions 7.* – 64 Bits

RHEL and CentOS versions 6.* – 64 Bits

RHEL and CentOS versions 6.* – 32 Bits

After install check if the repo is enabled

Now that we have it installed is time to use them

  • to use the EPEL repo you need to use the –-enablerepo option when running the yum package installer.

  • here we can see information about htop utility.

Next tutorial we will go over some basic yum commands.

 

 

 

Timezone data (tzdata) is stored in /usr/share/zoneinfo.

To change your CentOS server’s timezone, simply run the following command based up on the examples below:

 

 

Example of changing the timezone to Melbourne,Australia:

  • this will overwrite your actual local-time configuration.

What is Httpd or Apache2  ? 

Well is the same thing actually !

You can find Https on RedHat/CentOS/Fedora operational systems and Apache2 in Debian based distros such as Ubuntu.

Now that you know they the same let`s see how we can Restart their services in both occasions:

 

Https Service management options:

We can use service command:

Start Https Service

Restart Https Service

Stop Https Service

Another option is use /etc/init.d/httpd service script.

Start Https

Stop Https

Restart Https

 Apache2 Service management options:

We can use service command:

 

Another option is use /etc/init.d/apache2 service script.

 

Simple and easy tutorial but is a very common task.

 

What are SSH Keys ?

SSH Keys are based on the SSH cryptographic network protocol, which is responsible for the encryption of the information stream between you and the remote machine.

Why should we use SSH Keys ?

SSH keys provide a more secure way of logging into a virtual private server with SSH than using a password alone. While a password can eventually be cracked with a brute force attack, SSH keys are nearly impossible to decipher by brute force alone.

sshkeys

How does the SSH Key Login work ?

The SSH Key is a key pair that provides you with two long string of characters: a public and a private key. The public key is kept on the server that is then unlock it by connecting to it with a client that already has the private key. When the two match up, the system unlocks without the need for a password.

Is this safer than very complex text passwords ? 

Hell yeah, see using public key pairs offers considerably more protection than using passwords or password lists which can be captured if the client, the server or the secure session is compromised.

 

Now that you have an idea of how SSH Keys work and why should we use them lets see how to create and use them.

1  - Create the RSA Key Pair

The first step is to create the key pair on the client machine:

We need to use the ssh-keygen utility that comes shipped with openshh tools. If you dont have it installed follow this tutorial – Install OpenSSH.

  • after you have installed OpenSSH run the following command:

The output will be something like :

You can skip all questions and choose default values or you can provide your own options. I normally go with the default ones unless i have special passphrase i wanna add.

The public key is now located in /home/aodba/.ssh/id_rsa.pub and the private key (identification) is now located in /home/aodba/.ssh/id_rsa

2 – Copy/Use the generated Key.

So what do i mean by Copy/Use ?

Once the key pair is generated, it’s time to place the public key on the virtual server that we want to use. You can copy the public key into the new machine’s authorized_keys file.

  • so this will copy the key to the server you want to have access to your local(where the key was generated) server using the key.

3 – Test access using SSH Key

  •  from the Server(123.12.12.123) where you have copied the content of the id_rsa.pub into authorized_keys try to login into your initial server using the command bellow:

  • the “warning/message ” that you see is because is the first time you do a handshake using the key with this server. After this you will no longer see this message as the host will be in the known_host file (this file make verifies the server’s legitimacy).

Ok so setting up ssh key for Linux is done.

 

I hope this was helpful and if you have doubts or encounter any issues setting up ssh keys fell free to drop a comment or question and i will do my best to answer back or help you as much as i can,

 

This is the first chapter on BASH programming.

So, why should you learn bash programming?

Shouldn’t you learn a language that allows you to use and build upon these powerful time-saving constructs you already know how to use? Command shells unlock the potential of a UNIX system, and bash is the Linux shell. It’s the high-level glue between you and the machine. Grow in your knowledge of bash, and you’ll automatically increase your productivity under Linux and UNIX — it’s that simple.

So what are all this tutorials about ?

It is all about how to actually use bash programming constructs to be able to write your own stuff(and make sense out of it).

Learn the base so you can build a strong coding foundation and be able to use BASH scripting to optimize your work and time.

…i like how that sounds …

Check the next tutorial where we will initiate our journey into Bash Programming.

 

Environment variables in Bash Programming.

 

 

This tutorials are intended for Database Administrators and they don’t go into advanced Linux O.S. topics. Fell free to to take them as examples and as well use them, but remember the aodba.com will not be held responsible for any damages that you might experience using this tutorials.

Raid Level 5

     RAID 5 is by far the most common RAID configuration for business servers and enterprise NAS devices.
This RAID level provides better performance than mirroring as well as fault-tolerance. With RAID 5, data and parity (which is additional data used for recovery) are striped across three or more disks. Disk drives typically fail in sectors, rather than the entire drive dying.
When RAID 5 is configured, if a portion of a disk fails, that data gets recreated from the remaining data and parity, seamlessly and automatically.
This is beneficial because RAID 5 allows many NAS and server drives to be “hot-swappable” meaning in case a drive in the array fails, that drive can be swapped with a new drive without shutting down the server or NAS and without having to interrupt users who may be accessing the server or NAS.
RAID 5 can be implemented as a software or hardware solution.
The downside to RAID 5 is the performance hit to servers that perform a lot of write operations.

Graphical representation of the RAID layout:

Raid Level 5

Raid Level 5

Raid Level 4

Does not support multiple simultaneous write operations.
Stripes data at a block level across several drives, with parity stored on one drive.
The parity information allows recovery from the failure of any single drive.
The performance of a level 4 array is very good for reads (the same as level 0). However, writes require that parity data be updated each time. This slows small random writes, in particular, though large writes or sequential writes are fairly fast. The cost per megabyte of a RAID 4 array can be low because only one drive in the array stores redundant data.

Graphical representation of the RAID layout:

Raid Level 4

Raid Level 4

Raid Level 2

A RAID 2 stripes data at the bit (rather than block) level, and uses a Hamming code for error correction.

The disks are synchronized by the controller to spin at the same angular orientation (they reach Index at the same time), so it generally cannot service multiple requests simultaneously. Extremely high data transfer rates are possible. This is the only original level of RAID that is not currently used.

All hard disks eventually implemented Hamming code error correction. This made RAID 2 error correction redundant and unnecessarily complex. This level quickly became useless and is now obsolete. There are no commercial applications of RAID 2.

Graphical representation of the RAID layout:

Raid Level 2

Raid Level 2

Raid Level 0

RAID LEVEL 0
RAID 0 is used to boost a server’s performance.
It’s also known as “disk striping.” With RAID 0, data is written across multiple disks. This means the work that the computer is doing is handled by multiple disks rather than just one, increasing performance because multiple drives are reading and writing data, improving disk I/O. A minimum of two disks is required. Both software and hardware RAID support RAID 0 as do most controllers. The downside is that there is no fault tolerance. If one disk fails then that affects the entire array and the chances for data loss or corruption increases.

Graphical representation of the RAID layout:

Raid Level 0

Raid Level 0

RAID Levels

RAID is a storage technology that combines multiple disk drive components into a logical unit for the purposes of data redundancy and performance improvement. Data is distributed across the drives in one of several ways, referred to as RAID levels, depending on the specific level of redundancy and performance required.

 

raid

There are two types of RAID:

Hardware RAID 

Advantages and disadvantages with hardware-based RAID. It’s more expensive, because configuring it requires an additional hardware component, a RAID controller which is a piece of hardware that controls the RAID array. Hardware- based RAID is also considered a better performing, more efficient way to implement RAID than software RAID. Hardware-based RAID is used most in corporate servers and business-class NAS drives.

Software RAID

Software RAID is arguably not as reliable as hardware RAID, but it’s definitely more economical and can still deliver basic fault tolerance. You can’t configure RAID arrays as complex with software as you can with hardware, but if you just want to implement mirroring (which is copying data from one drive to another, to keep that data accessible in case a drive fails) then software RAID is a cheaper, less complicated to set up option. Instead of using a bunch of disks and a controller to make an array, some software RAID solutions can use logical partitions on a single disk. That’s what makes it both cheaper and less reliable if that single disk fails completely, your data is gone.

What level/type of RAID Is Right For Me?

Once you’ve decided whether software or hardware RAID best suits your purposes, you need to pick a RAID level this refers to how you are going to configure RAID on your device. There are several RAID levels, and the one you choose depends on whether you are using RAID for performance or fault tolerance (or both). It also matters whether you have hardware or software RAID, because software supports fewer levels than hardware-based RAID. In the case of hardware RAID, the type of controller you have matters, too. Different controllers support different levels of RAID and also dictate the kinds of disks you can use in an array: SAS, SATA or SSD).

List of RAID Levels:

To learn more about each RAID LEVEL just click on their links.

RAID LEVEL 0

Raid Level 0

Raid Level 0

RAID LEVEL 1

Raid Level 1

Raid Level 1

RAID LEVEL 2

Raid Level 2

Raid Level 2

RAID LEVEL 3

Raid Level 3

Raid Level 3

RAID LEVEL 4

Raid Level 4

Raid Level 4

RAID LEVEL 5

Raid Level 5

Raid Level 5

RAID LEVEL 6

Raid Level 6

Raid Level 6

RAID LEVEL 10

Raid Level 10

Raid Level 10

RAID LEVEL 50

Raid Level 50

Raid Level 50

To learn more about each RAID LEVEL just click on their links.