Here’s my recollections and evaluation of the Google IT Cert from coursera.org. This is Week 16, or, if you prefer, Course III, Week 4 – Filesystems.
. . .
Wow you’re making great progress now we’re going to continue doing other stuff this week we’re learning about filesystems and the tools you need to use disks.
Review of Filesystems
Remember when we were introduced to filesystems? No, neither do I. A filesystem is “used to keep track of files” on a disk. The filesystem is set so that an operating system knows how to find, read and write files on a disk.
The most popular filesystems in IT are NTFS on Windows and ext4 on Ubuntu Linux. You will usually find very little compatibility between operating systems and filesystems. For example, Ubuntu and Windows can both read and write to an NTFS drive, but only an Ubuntu machine can use an ext4 drive.
FAT32 is a filesystem that works with Windows, Linux, and MacOS, but it has some limitations: The maximum file size is 4GB, and the entire filesystem cannot exceed 32GB.
There is much more to come on compatibility, especially in the next course when we cover network filesystems.
Reading: FAT32 filesystems
This should be one of the most exciting documents of all time.
Disks can be set up with partitions, which are just pieces that can be managed, and allow the disk to behave as though you are using different physical disks. To add a filesystem to a disk, you must create a partition. A disk may have only one partition with one filesystem, or many partitions each with its own filesystem.
This practice of partitioning allows you to have one computer, one hard drive, two partitions and two different operating systems, for example.
It is important to note that once you format a partition with a filesystem, it becomes a volume. These terms are sometimes mixed up, but they are technically different and this is an important distinction.
A disk will also have a partition table, which tells the operating system how the disk has been partitioned. There are two main types of partition tables:
- Master Boot Record (MBR): Traditionally on Windows only, max 2TB volume size. Uses “primary partitions,” meaning that you can only have four partitions, unless you “extend” a primary partition to contain a logical The MBR is slowly being replaced.
- GUID Partition Table (GPT): 2TB (or greater) volume size. Only has one type of partition, and you may have unlimited partitions. UEFI BIOS requires that you are using the GUID partition table.
Windows: Disk Partitioning and Formatting a Filesystem
Windows has a built-in disk partitioning tool called the Disk Management Utility. You can right-click This PC, click Manage, then open the Disk Management under Storage. You can also hit the Windows key and start typing “disk management” and Windows will show you “create and format hard disk partitions”.
This tool shows you all the disks attached to the system. Probably a bad idea to learn about disk partitioning on the Windows partition, so maybe try a flash drive or an external drive.
You will see how a drive is formatted in Disk manager. To change the file system, right click on the partition and select Format. You can then select the volume label, or name of the disk, and the file system. Allocation Unit refers to the size of the block that will make up the partition. Generally, you will be okay selecting the default Allocation Unit size. You will also have an option to perform a “Quick Format” which skips checking the disk for errors. Finally, you can select to enable compression of files on the disk. Compression will slow down reading and writing files, but it will help you save disk space. You will probably not select this option for day-to-day disk usage. Just click OK to format the disk.
There is also a similar tool for the Windows CLI called Diskpart. Connect the disk you want to manage to the system and open cmd.exe. Then, open diskpart:
This will open a new terminal with the prompt “DISKPART”. List all the disks with the list disk command:
DISKPART> list disk Disk ### Status Size Free Dyn Gpt ----------- -------- ----- ------- ----- ----- Disk 0 Online 238 GB 0 B Disk 1 Online 14 GB 0 B DISKPART>
Say we want to re-partition a USB drive that is mounted as Disk 1. Start by selecting that disk, then wiping it of any partitions:
DISKPART> select disk 1 Disk 1 is now the selected disk. DISKPART> clean DIskPart succeeded in cleaning the disk. DISKPART> create partition primary DiskPart succeeded in creating the specified partition. DISKPART> select partition 1 Partition 1 is now the selected partition. DISKPART> active DiskPart marked the current partition as active. DISKPART> format FS-NTFS label-my-thumb-drive quick 100% completed DiskPart successfully formatted the volume. DISKPART>
That’s it. We just formatted a USB drive with the NTFS filesystem and gave it a volume label.
It’s always so helpful that the presenter reminds me I can watch the video again if I need to. Oh, just like every other video on the internet!
Reading: Windows Disk Partitioning and Formatting a Filesystem
Windows: Mounting and Unmounting a Filesystem
After connecting a drive to a system and formatting with the desired file system, the volume must be mounted in order to use it. Mounting a drive is a fun way of saying making it accessible.
Windows will do this automatically, as you have probably noticed when you connect a drive to your computer. USB drives are unmounted by right-clicking on the drive and selecting Eject.
Linux: Disk Partitioning and Formatting a Filesystem
There are a couple of tools in Linux we can use for partitioning drives. The Parted tool supports MBR and GPT partitioning, and can be used in two modes, interactive, which is run from a separate program, and command line, where you can just use the commands in the shell.
Let’s see what disks are connected to the system with the parted -l command:
user@user-home:~$ sudo parted -l
This will output the disk name, model, size, partition table type (MBR or GPT). In this example we have the line “Disk /dev/sda 128G” and in the list of partitions we have numbers 1, 2, and 3.
/dev/sda/1 is a 537MB partition formatted to FAT32. This partition starts at 1049KB and ends at 538MB.
We also have a /dev/sdb disk, but this has an “unknown” partition table, and a size of 7803MB. It is a Kingston DataTraveler 2.0 (scsi). I bet this is the USB drive.
Let’s run parted on the sdb disk:
user@user-home:~$ sudo parted /dev/sdb GNU Parted 3.2 Using /dev/sdb Welcome to GNU Parted: Type ‘help’ to view a list of commands. (parted)
Now we can run the print command to see the disk info again:
(parted) print Error: /dev/sdb unrecognized disk label Model: Kingston DataTraveler 2.0 (scsi) Disk: /dev/sdb: 7803MB Sector Size (logical/physical): 512b/512B Partition Table: unknown Disk Flags: (parted)
First, we need to label the disk
(parted) mklabel gpt (parted)
If we print again, we should see the partition table GPT:
(parted) print Model: Kingston DataTraveler 2.0 (scsi) Disk: /dev/sdb: 7803MB Sector Size (logical/physical): 512b/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags (parted)
Now, we’ll use the mkpart command, and include partition type, file system, start and end of disk:
(parted) mkpart primary ext4 1MiB 5GiB (Parted) print Model: Kingston DataTraveler 2.0 (scsi) Disk: /dev/sdb: 7803MB Sector Size (logical/physical): 512b/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049KB 5369MB 5368MB ext4 primary (parted)
Remember that the partition type is only important for MBR partitions, which use primary, extended and logical partitions. Because we are using GPT, we will just call it a primary.
Next, we format the partition using the mkfs command. But first, quit out of parted:
(parted) quit user@user-home:~$ sudo mkfs -t ext4 /dev/sdb1
That creates an ext4 filesystem on /dev/sdb partition 1.
Remember to be careful while partitioning disks!
Next, mount that disk so it can be used.
Linux: Mounting and Unmounting a Filesystem
After partitioning and formatting a disk, this is the final step before being able to use a disk.
In short, you can’t just cd into a disk. You have to create a directory on the machine and mount the filesystem to that directory. The partition we want to access is /dev/sdb1.
We have a directory in root called “my_usb”.
user@user-home:~$ sudo mount /dev/sdb1 /my_usb/ user@user-home:~$ cd /my_usb? user@user-home:/my_usb$
In real life, most operating systems will automatically mount and unmount USB devices, but you need to know this for regular drives.
You can unmount a filesystem with the umount command:
user@user-home:~$ sudo umount /my_usb/ user@user-home:~$
Or you can unmount the actual device:
user@user-home:~$ sudo umount /dev/sdb1 user@user-home:~$
Shutting down the computer automatically unmounts the drives, referred to here as “removing the mount points.” Be sure to unmount drives before physically disconnecting and removing them from the system.
You can permanently mount a disk, so that it will mount when the system boots, by modifying a file called /etc/fstab.
user@user-home:~$ cat /etc/fstab
Will show us entries for devices. To make a new entry, we first need to find the UUID of the device:
user@user-home:~$ sudo blkid
This outputs a list of /dev/sda and sdb and sdc UUID’s, but our presenter just stops explaining how to make the entry into /etc/fstab. What was the point???
Reading: Linux Mounting and Unmounting a Filesystem
Read more on mounting and unmounting filesystems in Linux.
Swap Space is a computing term that I have heard but am not familiar with. To understand swap space, we need to be aware of the concept of virtual memory.
Virtual memory is “how our OS provides the physical memory available in our computer (like RAM) to the applications that run on the computer”. I can’t really say that definition helped me understand the concept. The OS translates virtual addresses to physical addresses, so that software can use RAM? I think?
This means that the system can use more space than is actually provided by RAM, because the OS can allocate some hard drive space to be used as storage for pages that have been evicted from RAM because they were not in use. But, when a program needs that data, it is still available as virtual memory, located on the hard drive.
Windows uses a program called Memory Manager to handle virtual memory. This program does the mapping from RAM to hdd, and saves those pages on a special, hidden file in the root partition of the volume called pagefile.sys.
Windows will let you control paging configuration through the Control Panel using an applet called System Properties. Open Control Panel, go to System and Security, and click on System. Then open the Advanced System Settings item on the left. Under the Advanced tab click the Setting button, and in that Performance Options box you can click the Advanced tab and see the option to change the Virtual Memory settings. Ona 64-bit Windows system, this should be set to 1x the amount of installed RAM. It is usually fine to leave this setting as-is. (Personally, I would avoid playing with this unless you are having a problem, which, because it is Windows, is just a matter of time.)
Reading: Windows Paging
Swap space in Linux is the area of the hard drive used for virtual memory. Back to our USB drive—we’re going to use the rest of the space as swap space. Go back into the parted tool and select the USB drive, at /dev/sdb
user@user-home:/$ sudo parted /dev/sdb Model: Kingston DataTraveler 2.0 (scsi) Disk: /dev/sdb: 7803MB Sector Size (logical/physical): 512b/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049KB 5369MB 5368MB ext4 primary (parted) mkpart primary linux-swap 5GiB 100% (parted) print Model: Kingston DataTraveler 2.0 (scsi) Disk: /dev/sdb: 7803MB Sector Size (logical/physical): 512b/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049KB 5368MB 5368MB ext4 primary 2 5369KB 7802MB 2434MB linux-swap v1 primary (parted)
This isn’t done – “swap” is not a filesystem. We need to exit parted and use the mkswap command on this new swap partition:
user@user-home:/$ sudo mkswap /dev/sdb2
That just set it up, now turn it on using the swapon command:
user@user-home:/$ sudo swapon /dev/sdb2
You can also automatically set up swap space on boot up by creating a swap entry on /etc/fstab, even though the video didn’t explain how to do that. Thanks.
Reading: Linux Swap
Here is more on swap in Linux systems.
You’ve probably heard a lot about files. Files contain data and metadata. Here, we’re going to go over what those are.
File Data refers to the actual data of a document, such as the words in a text document. The file metadata refers to all that information about the file, like the owner, data modified, permissions, etc.
Remember that NTFS is the native file system of our Windows system. How does NTFS store and interact with files? It uses a Master File Table, or MFT. Every file in a volume has an entry in the MFT (including the MFT itself). There is usually one MFT record for each file, but files with many attributes may have more than one MFT record. The MFT entry will include information like file name, timestamp, permissions, compression, location, etc.
When you create files on an NTFS system, entries are created in the MFT. When files are deleted, their entries are marked as “free” and then can be reused.
A file has an important identifier in the MFT called a File Record Number.
Shortcuts are files that redirect the system to another location, and they have an MFT entry just like any other file. Create a shortcut by right-clicking a file and selecting Create Shortcut.
Symbolic Links are like shortcuts, but they are working at the file system level. A symbolic link in the MFT will point to another entry or another file. The OS treats symbolic links the same as real files in almost every respect.
Make a folder on the desktop called “Links” and a text file in that folder called “File_1”, and write the word “Hello” in the file. Then make a shortcut to that file called “File_1_Shortcut”.
Now, open a command prompt and go to the Links directory:
C:\Users\user\Desktop> cd Links C:\Users\user\Desktop\Links>
Try to open the file using the shortcut through the notepad program:
C:\Users\user\Desktop\Links> notepad.exe file_1_shortcut.lnk
This doesn’t work, we only see the contents of the shortcut file, not the contents of the file the shortcut points to.
Try using a symbolic link, by creating one using the mklink program from the command prompt, and then open that with notepad:
C:\Users\user\Desktop\Links> mklink file_1_symlink file_1.txt Symbolic link created for file_1_symlink < <- - -> > file_1.txt C:\Users\user\Desktop\Links> notepad.exe file_1_symlink
It worked, we opened the original file and see the word “hello”.
A hard link is similar to a symbolic link, but it points to the linked file record number, not the file name. This means that a filename can change and the hard link will still function. You can make a hard link using the /h option:
C:\Users\user\Desktop\Links> mklink /h file_1_hardlink file_1.txt
Reading: NTFS Filesystem
Metadata and files are structured into an inode, which is similar to the MFT record in Windows NTFS. Inodes are stored in an inode table. The inode itself does not store any file data or file names, just everything else about the file.
Linux operates in the same conceptual way as Windows does in regards to softlinks (link symbolic links in Windows) which point to file names. Hardlinks don’t point to file names, but rather point to an inode in the inode table.
If you run
user@user-machine:/Desktop$ ls -l important_file -rw-rw-r—1 user user 0 Jan 5 15:43 important_file
You notice the third field is a 1, indicating that this file has one hardlink. When the hardlink count is 0, it means the file has been completely removed from the system.
Create a softlink using the command ln -s:
user@user-machine:/Desktop$ ln -s important_file important_file_softlink user@user-machine:/Desktop$ ls important_file important_file_softlink user@user-machine:/Desktop$
Create a hardlink using the ln command without the -s:
user@user-machine:/Desktop$ ln important_file important_file_hardlink important_file important_file_softlink important_file_hardlink user@user-machine:/Desktop$
Hardlinks allow you to store a file in multiple places without taking up more space on the volume.
Windows: Disk Usage
We can use the Disk Management console inside the Computer Management utility to monitor disk usage. Right-click on a partition and select Properties to view disk information for that partition. (There is also a command line utility called Disk Usage as part of the sysinternals package. Reading to follow.)
The General tab has a button to initiate cleanmgr.exe, which is a tool that can free up space by deleting temp files, compressing unused files, and emptying the recycle bin. Defragmentation can be used to re-organize data so that rotational hard drives can read and write faster. SSD’s do not operate that way, but there is a tool called trim that can improve performance. Windows 10 automates defragmentation, but you can also run it manually from the Disk Defragmenter tool in Windows.
Reading: Windows Disk Usage
Linux: Disk Usage
Use the du -h command to view disk utilization in Linux, you dummy.
user@user-machine:/$ du -h
This will output a lot of stuff. The -h option renders the output in human-readable form. Us the df command to view the amount of free space:
user@user-machine:/$ df -h
Linux generally does a good job handling fragmentation, so we are not going to discuss that.
Reading: Disk Usage
But you can read all about Linux not needing defragmentation.
Windows: Filesystem Repair
Remember when you unplugged that USB device without ejecting first? And remember the scolding error saying you must eject the drive first? It may seem like you were done writing to the drive, but because of the way the system works that may not be true.
A data buffer is a region of RAM used to store data temporarily while it is moved around. If the USB drive is unplugged before the data has been fully written from the buffer it can cause corruption. Other failures, like power-loss and software problems, can cause corruption.
Windows NTFS uses a system called journaling which logs changes to files and file metadata. If there is a system crash resulting in corruption, the OS can review the log and re-write parts of files to the state they were in before the corruption.
There is a self-healing tool running in Windows that can repair files, fsutil, check it out in an admin cmd like this:
C:\Windows\system32> fsutil repair query C: Self healing state on C: is: 0x9 Values: 0x1 - Enable general repair 0x9 - Enable repair and warn about potential data loss 0x10 - Disable repair and bugcheck once on first corruption C:\Windows\system32>
Once you have sever disk problems, like bad sectors and disk failures, you can use the NTFS chkdsk utility. Use an admin cmd and use the chkdsk command with the /f flag to fix errors it finds. You can also specify the drive to check:
C:\Windows\system32> chkdsk /f D:
This will output a lot of status info.
Chkdsk will often run automatically if Windows detects any severe corruption when it boots.
Linux: Filesystem Repair
Linux has the fsck command to check filesystems. MAKE SURE THE VOLUME IS UNMOUNTED. Running fsck on a mounted filesystem will likely damage it.
user@user-machine:/$ sudo fsck /dev/sda
Some versions of Linux will run fsck when they boot in an attempt to repair any issues automatically. There’s more to read about up next!
Reading: Linux Filesystem Repair
Read up on fsck here.
Ben like computer. Ben like solve problem on machine. Ben make two minute video explain he like computer as child.
Have I ever used a disk partitioning tool? What did I use it for? Why don’t I get credit for answering the discussion questions? So many questions.
I passed the quiz but I seem to miss when a question is asking for multiple answers. I get the ones I mark correct, but I should read the questions better. I’m losing a lot of points here, Jerry!
One of the least painful assessments – formatting and partitioning drives in Windows and Linux.
Great job, everybody. See you next week for week 15.
Samsung 860 EVO 500GB 2.5 Inch SATA III Internal SSD World Without Mind: The Existential Threat of Big Tech | by Franklin Foer Live Work Work Work Die: A Journey into the Savage Heart of Silicon Valley | Corey Pein Lenovo ThinkPad Edge E470 14" HD 1366x768 i5-6200U 2.3 GHz, 8GB 240GB SSD Windows 10 Pro The Hapless Rube's Apocalypse Survival Guide | by Jack Barker