Here’s my recollections and evaluation of the Google IT Cert from coursera.org. This is Week 16, or, if you prefer, Course III, Week 4 – Filesystems.

.   .   .

 

Module Intro

Wow you’re making great progress now we’re going to continue doing other stuff this week we’re learning about filesystems and the tools you need to use disks.

Review of Filesystems

Remember when we were introduced to filesystems? No, neither do I. A filesystem is “used to keep track of files” on a disk. The filesystem is set so that an operating system knows how to find, read and write files on a disk.

The most popular filesystems in IT are NTFS on Windows and ext4 on Ubuntu Linux. You will usually find very little compatibility between operating systems and filesystems. For example, Ubuntu and Windows can both read and write to an NTFS drive, but only an Ubuntu machine can use an ext4 drive.

FAT32 is a filesystem that works with Windows, Linux, and MacOS, but it has some limitations: The maximum file size is 4GB, and the entire filesystem cannot exceed 32GB.

There is much more to come on compatibility, especially in the next course when we cover network filesystems.

Reading: FAT32 filesystems

This should be one of the most exciting documents of all time.

Disk Anatomy

Disks can be set up with partitions, which are just pieces that can be managed, and allow the disk to behave as though you are using different physical disks. To add a filesystem to a disk, you must create a partition. A disk may have only one partition with one filesystem, or many partitions each with its own filesystem.

This practice of partitioning allows you to have one computer, one hard drive, two partitions and two different operating systems, for example.

It is important to note that once you format a partition with a filesystem, it becomes a volume. These terms are sometimes mixed up, but they are technically different and this is an important distinction.

A disk will also have a partition table, which tells the operating system how the disk has been partitioned. There are two main types of partition tables:

• Master Boot Record (MBR): Traditionally on Windows only, max 2TB volume size. Uses “primary partitions,” meaning that you can only have four partitions, unless you “extend” a primary partition to contain a logical The MBR is slowly being replaced.
• GUID Partition Table (GPT): 2TB (or greater) volume size. Only has one type of partition, and you may have unlimited partitions. UEFI BIOS requires that you are using the GUID partition table.

Windows: Disk Partitioning and Formatting a Filesystem

Windows has a built-in disk partitioning tool called the Disk Management Utility. You can right-click This PC, click Manage, then open the Disk Management under Storage. You can also hit the Windows key and start typing “disk management” and Windows will show you “create and format hard disk partitions”.

This tool shows you all the disks attached to the system. Probably a bad idea to learn about disk partitioning on the Windows partition, so maybe try a flash drive or an external drive.

You will see how a drive is formatted in Disk manager. To change the file system, right click on the partition and select Format. You can then select the volume label, or name of the disk, and the file system. Allocation Unit refers to the size of the block that will make up the partition. Generally, you will be okay selecting the default Allocation Unit size. You will also have an option to perform a “Quick Format” which skips checking the disk for errors. Finally, you can select to enable compression of files on the disk. Compression will slow down reading and writing files, but it will help you save disk space. You will probably not select this option for day-to-day disk usage. Just click OK to format the disk.

There is also a similar tool for the Windows CLI called Diskpart. Connect the disk you want to manage to the system and open cmd.exe. Then, open diskpart:

C:\Windows\system32> diskpart

This will open a new terminal with the prompt “DISKPART”. List all the disks with the list disk command:

DISKPART> list disk
            Disk ###          Status   Size        Free      Dyn        Gpt
            -----------       --------   -----     -------   -----        -----
            Disk 0             Online   238 GB      0 B
            Disk 1             Online   14 GB       0 B
DISKPART>

Say we want to re-partition a USB drive that is mounted as Disk 1. Start by selecting that disk, then wiping it of any partitions:

DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> clean
DIskPart succeeded in cleaning the disk.
DISKPART> create partition primary
DiskPart succeeded in creating the specified partition.
DISKPART> select partition 1
Partition 1 is now the selected partition.
DISKPART> active
DiskPart marked the current partition as active.
DISKPART> format FS-NTFS label-my-thumb-drive quick
                100% completed
DiskPart successfully formatted the volume.
DISKPART>

That’s it. We just formatted a USB drive with the NTFS filesystem and gave it a volume label.

It’s always so helpful that the presenter reminds me I can watch the video again if I need to. Oh, just like every other video on the internet!

Reading: Windows Disk Partitioning and Formatting a Filesystem

Readings here about setting allocation unit sizes and the DiskPart command.

 Windows: Mounting and Unmounting a Filesystem

After connecting a drive to a system and formatting with the desired file system, the volume must be mounted in order to use it. Mounting a drive is a fun way of saying making it accessible.

Windows will do this automatically, as you have probably noticed when you connect a drive to your computer. USB drives are unmounted by right-clicking on the drive and selecting Eject.

Linux: Disk Partitioning and Formatting a Filesystem

There are a couple of tools in Linux we can use for partitioning drives. The Parted tool supports MBR and GPT partitioning, and can be used in two modes, interactive, which is run from a separate program, and command line, where you can just use the commands in the shell.

Let’s see what disks are connected to the system with the parted -l command:

user@user-home:~$ sudo parted -l

This will output the disk name, model, size, partition table type (MBR or GPT). In this example we have the line “Disk /dev/sda 128G” and in the list of partitions we have numbers 1, 2, and 3.

/dev/sda/1 is a 537MB partition formatted to FAT32. This partition starts at 1049KB and ends at 538MB.

We also have a /dev/sdb disk, but this has an “unknown” partition table, and a size of 7803MB. It is a Kingston DataTraveler 2.0 (scsi). I bet this is the USB drive.

Let’s run parted on the sdb disk:

user@user-home:~$ sudo parted /dev/sdb
GNU Parted 3.2
Using /dev/sdb
Welcome to GNU Parted: Type ‘help’ to view a list of commands.
(parted)

Now we can run the print command to see the disk info again:

(parted) print
Error: /dev/sdb unrecognized disk label
Model: Kingston DataTraveler 2.0 (scsi)
Disk: /dev/sdb: 7803MB
Sector Size (logical/physical): 512b/512B
Partition Table: unknown
Disk Flags:
(parted)

First, we need to label the disk

(parted) mklabel gpt
(parted)

If we print again, we should see the partition table GPT:

(parted) print
Model: Kingston DataTraveler 2.0 (scsi)
Disk: /dev/sdb: 7803MB
Sector Size (logical/physical): 512b/512B
Partition Table: gpt
Disk Flags:
Number               Start      End        Size        File system          Name    Flags
(parted)

Now, we’ll use the mkpart command, and include partition type, file system, start and end of disk:

(parted) mkpart primary ext4 1MiB 5GiB
(Parted) print
Model: Kingston DataTraveler 2.0 (scsi)
Disk: /dev/sdb: 7803MB
Sector Size (logical/physical): 512b/512B
Partition Table: gpt
Disk Flags:
Number    Start   End      Size        File system    Name         Flags
1         1049KB  5369MB   5368MB      ext4           primary
(parted)

Remember that the partition type is only important for MBR partitions, which use primary, extended and logical partitions. Because we are using GPT, we will just call it a primary.

Next, we format the partition using the mkfs command. But first, quit out of parted:

(parted) quit

user@user-home:~$ sudo mkfs -t ext4 /dev/sdb1

That creates an ext4 filesystem on /dev/sdb partition 1.

Remember to be careful while partitioning disks!

Next, mount that disk so it can be used.

Linux: Mounting and Unmounting a Filesystem

After partitioning and formatting a disk, this is the final step before being able to use a disk.

In short, you can’t just cd into a disk. You have to create a directory on the machine and mount the filesystem to that directory. The partition we want to access is /dev/sdb1.

We have a directory in root called “my_usb”.

user@user-home:~$ sudo mount /dev/sdb1 /my_usb/
user@user-home:~$ cd /my_usb?
user@user-home:/my_usb$

In real life, most operating systems will automatically mount and unmount USB devices, but you need to know this for regular drives.

You can unmount a filesystem with the umount command:

user@user-home:~$ sudo umount /my_usb/
user@user-home:~$

Or you can unmount the actual device:

user@user-home:~$ sudo umount /dev/sdb1
user@user-home:~$

Shutting down the computer automatically unmounts the drives, referred to here as “removing the mount points.” Be sure to unmount drives before physically disconnecting and removing them from the system.

You can permanently mount a disk, so that it will mount when the system boots, by modifying a file called /etc/fstab.

user@user-home:~$ cat /etc/fstab

Will show us entries for devices. To make a new entry, we first need to find the UUID of the device:

user@user-home:~$ sudo blkid

This outputs a list of /dev/sda and sdb and sdc UUID’s, but our presenter just stops explaining how to make the entry into /etc/fstab. What was the point???

Reading: Linux Mounting and Unmounting a Filesystem

Read more on mounting and unmounting filesystems in Linux.

Windows: Swap

Swap Space is a computing term that I have heard but am not familiar with. To understand swap space, we need to be aware of the concept of virtual memory.

Virtual memory is “how our OS provides the physical memory available in our computer (like RAM) to the applications that run on the computer”. I can’t really say that definition helped me understand the concept. The OS translates virtual addresses to physical addresses, so that software can use RAM? I think?

This means that the system can use more space than is actually provided by RAM, because the OS can allocate some hard drive space to be used as storage for pages that have been evicted from RAM because they were not in use. But, when a program needs that data, it is still available as virtual memory, located on the hard drive.

Windows uses a program called Memory Manager to handle virtual memory. This program does the mapping from RAM to hdd, and saves those pages on a special, hidden file in the root partition of the volume called pagefile.sys.

Windows will let you control paging configuration through the Control Panel using an applet called System Properties. Open Control Panel, go to System and Security, and click on System. Then open the Advanced System Settings item on the left. Under the Advanced tab click the Setting button, and in that Performance Options box you can click the Advanced tab and see the option to change the Virtual Memory settings. Ona 64-bit Windows system, this should be set to 1x the amount of installed RAM. It is usually fine to leave this setting as-is. (Personally, I would avoid playing with this unless you are having a problem, which, because it is Windows, is just a matter of time.)

Reading: Windows Paging

Read up on paging in Windows and determining the appropriate page file size for your Windows system.

Linux: Swap

Swap space in Linux is the area of the hard drive used for virtual memory. Back to our USB drive—we’re going to use the rest of the space as swap space. Go back into the parted tool and select the USB drive, at /dev/sdb

user@user-home:/$ sudo parted /dev/sdb
Model: Kingston DataTraveler 2.0 (scsi)
Disk: /dev/sdb: 7803MB
Sector Size (logical/physical): 512b/512B
Partition Table: gpt
Disk Flags:
Number    Start   End      Size        File system    Name         Flags
1         1049KB  5369MB   5368MB      ext4           primary

(parted) mkpart primary linux-swap 5GiB 100%
(parted) print
Model: Kingston DataTraveler 2.0 (scsi)
Disk: /dev/sdb: 7803MB
Sector Size (logical/physical): 512b/512B
Partition Table: gpt
Disk Flags:
Number    Start   End      Size        File system     Name       Flags
1         1049KB  5368MB   5368MB      ext4            primary
2         5369KB  7802MB   2434MB      linux-swap v1   primary
(parted)

This isn’t done – “swap” is not a filesystem. We need to exit parted and use the mkswap command on this new swap partition:

(parted) quit

user@user-home:/$ sudo mkswap /dev/sdb2

That just set it up, now turn it on using the swapon command:

user@user-home:/$ sudo swapon /dev/sdb2

You can also automatically set up swap space on boot up by creating a swap entry on /etc/fstab, even though the video didn’t explain how to do that. Thanks.

Reading: Linux Swap

Here is more on swap in Linux systems.

 Windows: Files

You’ve probably heard a lot about files. Files contain data and metadata. Here, we’re going to go over what those are.

File Data refers to the actual data of a document, such as the words in a text document. The file metadata refers to all that information about the file, like the owner, data modified, permissions, etc.

Remember that NTFS is the native file system of our Windows system. How does NTFS store and interact with files? It uses a Master File Table, or MFT. Every file in a volume has an entry in the MFT (including the MFT itself). There is usually one MFT record for each file, but files with many attributes may have more than one MFT record. The MFT entry will include information like file name, timestamp, permissions, compression, location, etc.

When you create files on an NTFS system, entries are created in the MFT. When files are deleted, their entries are marked as “free” and then can be reused.

A file has an important identifier in the MFT called a File Record Number.

Shortcuts are files that redirect the system to another location, and they have an MFT entry just like any other file. Create a shortcut by right-clicking a file and selecting Create Shortcut.

Symbolic Links are like shortcuts, but they are working at the file system level. A symbolic link in the MFT will point to another entry or another file. The OS treats symbolic links the same as real files in almost every respect.

Make a folder on the desktop called “Links” and a text file in that folder called “File_1”, and write the word “Hello” in the file. Then make a shortcut to that file called “File_1_Shortcut”.

Now, open a command prompt and go to the Links directory:

C:\Users\user\Desktop> cd Links
C:\Users\user\Desktop\Links>

Try to open the file using the shortcut through the notepad program:

C:\Users\user\Desktop\Links> notepad.exe file_1_shortcut.lnk

This doesn’t work, we only see the contents of the shortcut file, not the contents of the file the shortcut points to.

Try using a symbolic link, by creating one using the mklink program from the command prompt, and then open that with notepad:

C:\Users\user\Desktop\Links> mklink file_1_symlink file_1.txt
Symbolic link created for file_1_symlink < <- - -> > file_1.txt
C:\Users\user\Desktop\Links> notepad.exe file_1_symlink

It worked, we opened the original file and see the word “hello”.

A hard link is similar to a symbolic link, but it points to the linked file record number, not the file name. This means that a filename can change and the hard link will still function. You can make a hard link using the /h option:

C:\Users\user\Desktop\Links> mklink /h file_1_hardlink file_1.txt

Reading: NTFS Filesystem

Readings on Master File Table, Creating Symbolic Links, and Hard Links and Junctions.

 Linux: Files

Metadata and files are structured into an inode, which is similar to the MFT record in Windows NTFS. Inodes are stored in an inode table. The inode itself does not store any file data or file names, just everything else about the file.

Linux operates in the same conceptual way as Windows does in regards to softlinks (link symbolic links in Windows) which point to file names. Hardlinks don’t point to file names, but rather point to an inode in the inode table.

If you run

user@user-machine:/Desktop$ ls -l important_file
-rw-rw-r—1 user user 0 Jan 5 15:43 important_file

You notice the third field is a 1, indicating that this file has one hardlink. When the hardlink count is 0, it means the file has been completely removed from the system.

Create a softlink using the command ln -s:

user@user-machine:/Desktop$ ln -s important_file important_file_softlink
user@user-machine:/Desktop$ ls
important_file important_file_softlink
user@user-machine:/Desktop$

Create a hardlink using the ln command without the -s:­

user@user-machine:/Desktop$ ln  important_file important_file_hardlink
important_file important_file_softlink important_file_hardlink
user@user-machine:/Desktop$

Hardlinks allow you to store a file in multiple places without taking up more space on the volume.

Windows: Disk Usage

We can use the Disk Management console inside the Computer Management utility to monitor disk usage. Right-click on a partition and select Properties to view disk information for that partition. (There is also a command line utility called Disk Usage as part of the sysinternals package. Reading to follow.)

The General tab has a button to initiate cleanmgr.exe, which is a tool that can free up space by deleting temp files, compressing unused files, and emptying the recycle bin. Defragmentation can be used to re-organize data so that rotational hard drives can read and write faster. SSD’s do not operate that way, but there is a tool called trim that can improve performance. Windows 10 automates defragmentation, but you can also run it manually from the Disk Defragmenter tool in Windows.

Reading: Windows Disk Usage

Read about Disk Usage and starting Disk Cleanup from the command line.

Linux: Disk Usage

Use the du -h command to view disk utilization in Linux, you dummy.

user@user-machine:/$ du -h

This will output a lot of stuff. The -h option renders the output in human-readable form. Us the df command to view the amount of free space:

user@user-machine:/$ df -h

Linux generally does a good job handling fragmentation, so we are not going to discuss that.

Reading: Disk Usage

But you can read all about Linux not needing defragmentation.

Windows: Filesystem Repair

Remember when you unplugged that USB device without ejecting first? And remember the scolding error saying you must eject the drive first? It may seem like you were done writing to the drive, but because of the way the system works that may not be true.

A data buffer is a region of RAM used to store data temporarily while it is moved around. If the USB drive is unplugged before the data has been fully written from the buffer it can cause corruption. Other failures, like power-loss and software problems, can cause corruption.

Windows NTFS uses a system called journaling which logs changes to files and file metadata. If there is a system crash resulting in corruption, the OS can review the log and re-write parts of files to the state they were in before the corruption.

There is a self-healing tool running in Windows that can repair files, fsutil, check it out in an admin cmd like this:

C:\Windows\system32> fsutil repair query C:
Self healing state on C: is: 0x9
Values: 0x1  - Enable general repair
        0x9  - Enable repair and warn about potential data loss
        0x10 - Disable repair and bugcheck once on first corruption
C:\Windows\system32>

Once you have sever disk problems, like bad sectors and disk failures, you can use the NTFS chkdsk utility. Use an admin cmd and use the chkdsk command with the /f flag to fix errors it finds. You can also specify the drive to check:

C:\Windows\system32> chkdsk /f D:

This will output a lot of status info.

Chkdsk will often run automatically if Windows detects any severe corruption when it boots.

Linux: Filesystem Repair

Linux has the fsck command to check filesystems. MAKE SURE THE VOLUME IS UNMOUNTED. Running fsck on a mounted filesystem will likely damage it.

user@user-machine:/$ sudo fsck /dev/sda

Some versions of Linux will run fsck when they boot in an attempt to repair any issues automatically. There’s more to read about up next!

Reading: Linux Filesystem Repair

Read up on fsck here.

“Ben Passion”

Ben like computer. Ben like solve problem on machine. Ben make two minute video explain he like computer as child.

>>>discussion

Have I ever used a disk partitioning tool? What did I use it for? Why don’t I get credit for answering the discussion questions? So many questions.

>>>Quiz

I passed the quiz but I seem to miss when a question is asking for multiple answers. I get the ones I mark correct, but I should read the questions better. I’m losing a lot of points here, Jerry!

>>>>>>>Assessments

One of the least painful assessments – formatting and partitioning drives in Windows and Linux.

Great job, everybody. See you next week for week 15.

---

Samsung 860 EVO 500GB 2.5 Inch SATA III Internal SSD

World Without Mind: The Existential Threat of Big Tech | by Franklin Foer

Live Work Work Work Die: A Journey into the Savage Heart of Silicon Valley | Corey Pein

Lenovo ThinkPad Edge E470 14" HD 1366x768 i5-6200U 2.3 GHz, 8GB 240GB SSD Windows 10 Pro 

The Hapless Rube's Apocalypse Survival Guide | by Jack Barker