Deleting relatively large files in the WSL, such as Docker caches and images, has not resulted in any reduction in space from the host Windows perspective. In particular, deleting the Dcoker version of the stable diffusion model has no effect at all. Therefore, we will explain the problem that deleting files in WSL (Windows Subsystem for Linux) does not reduce the disk space of the host.
In WSL, the Linux file system exists virtually on the Windows file system, and while there are slight differences in this behavior between WSL 1 and WSL 2, basically deleting a file within WSL may not immediately free up disk space on the Windows side. This is due to the way WSL manages the virtual disk it uses.
In the case of WSL 2
WSL 2 uses the actual Linux kernel and the file system is stored in a VHD (Virtual Hard Disk) file. Deleting files does not automatically reduce the size of the VHD; to reduce the VHD file size, you must manually compress the disk. To do this, you can temporarily stop the WSL instance and run the Optimize-VHD command in PowerShell. even in VMware, uninstalling applications and other files in the guest OS did not reduce the size, so it was compressed. In this case, it was relatively easy because there is a GUI tool.
In case of WSL 1
In WSL 1, the file system is integrated directly within the Windows file system. But even here, you may not see immediate freeing of disk space after a file is deleted. This is because the Windows file system does not immediately reuse space for deleted files.
Solution
- Disk Comp ression: If you want to reduce the size of VHD files in WSL 2, you can try disk compression.
- Delete temporary files: You can use the
sudo apt-get cleanorrmcommand within WSL to delete unnecessary files and caches to free up some disk space. - Restarting WSL: Especially in WSL 1, restarting WSL may allow Windows to clean up the file system and regain disk space.
However, the Optimize-VHD command may cause an error.
PS C:\Windows\System32> Optimize-VHD Optimize-VHD: The term ‘Optimize-VHD’ is not recognized as a name of a cmdlet, function, script file, or Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
If you receive an error message that the Optimize-VHD command is not recognized, the Hyper-V PowerShell module may not be installed or loaded properly. Optimize-VHD is part of the PowerShell cmdlet for managing Hyper-V features and is available on Windows systems with Hyper-V features enabled.
Solution.
- Check for and enable Hyper-V functionality: Hyper-V must be enabled on Windows in order to use
Optimize-VHD. However, Hyper-V is only available on Windows 10 Pro, Education, or Enterprise editions; it is not available on Windows 10 Home editions.- Open the Control Panel.
- Go to Programs and Features and select Enable or Disable Windows Features.
- Find Hyper-V in the list and check the box.
- Click OK and follow the instructions to restart your computer.
- Install Hyper-V PowerShell Module: If Hyper-V features are enabled but the
Optimize-VHDcommand is not available, the Hyper-V PowerShell module may not be installed. You can install the Hyper-V PowerShell module using the following commandInstall-WindowsFeature -Name Hyper-V-PowerShell
PowerShell must be opened with administrative privileges to execute the above command. - Restart PowerShell: After installing the module, restart your PowerShell session so that the newly installed cmdlets are recognized.
Here is how to reduce it in the HOME edition.
It is possible to reduce the physical size of the dynamically expanding virtual hard disk (VHD) file using the “diskpart compact vdisk command”. This method is used to compress the unused space on the VHD and reduce the actual disk space occupied. This command is particularly useful for managing the size of VHD files used in WSL 2.
How to use
- Open PowerShell or a command prompt as an administrator.
- Run
diskpartto launch the DiskPart tool. - At the DiskPart prompt, use the command
select vdisk file="file path"to select the target VHD file. The “file path” here is the full path of the VHD file you wish to shrink. - Run the
compact vdiskcommand to reduce the size of the VHD file.
Notes
- Importance of Backups: Before performing any disk operation, it is important to make a backup of any data that may be lost.
- Shutting down WSL: The WSL instance must be shut down before executing the
compact vdiskcommand. This is done using thewsl --shutdowncommand. It is important to shut down the guest OS. The disk shrink operation cannot be performed while the guest OS is running. - Target VHD file: The location of the VHD file used by WSL 2 is usually in the folder associated with
$HOME\AppData\Local\Packagesin the Windows user profile directory. The exact path will vary depending on the Linux distribution installed.
The command prompt (cmd.exe) differs from PowerShell in the way it references environment variables: while in PowerShell you can refer to the path to the user home directory with the notation $HOME or${HOME}, the command prompt does the same using %USERPROFILE% to perform the same operation. Navigate to the AppData\Local\Packages directory in the current user’s profile directory. The %USERPROFILE% represents the path to the home directory of the currently logged in user (usually C:\Users\<Username>).
About shutdown.
It is possible, and in some cases recommended, to shut down directly from within the guest OS; if you are using WSL (Windows Subsystem for Linux), simply closing the window of the Linux distribution (e.g. Ubuntu), background processes may continue to run. This is because the WSL session remains active in the background and may continue to consume resources.
How to Shut Down in WSL
- Using the shutdown command within the guest OS: You can use the
shutdowncommand in Linux distributions such as Ubuntu. To perform an immediate shutdown, enter the following command in a terminal
sudo shutdown now This command safely terminates the current session and shuts down the Linux system. This is the method I usually use; you can also shut down the entire WSL instance from a Windows command prompt or PowerShell. This is the way to end all Linux sessions and stop the WSL service. The previously mentioned wsl --shutdown.
File Locations at the Command Prompt%USERPROFILE%\AppData\Local\Packages
Location of the file in PowerShell$HOME\AppData\Local\Packages
If you are using WSL (Windows Subsystem for Linux), data for installed Linux distributions (e.g., Ubuntu) is stored in a specific location in the Windows current user profile directory. Specifically, it is located in the AppData\Local\Packages directory.
For Ubuntu, that data is stored in a folder named CanonicalGroupLimited.UbuntuonWindows... The data is stored in a folder with a name like “CanonicalGroupLimited. The folder name may vary depending on the version of Ubuntu and WSL installed. This folder contains the file system of the Linux environment, and by accessing the virtual hard disk files (files with .vhdx extension) in the LocalState subfolder, you can directly access the data of the Linux distribution of WSL For example, you can access Ubuntu in WSL.
For example, if you are using Ubuntu with WSL, the path to the virtual disk could be as follows
C:\Users\YourUsername\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows... \LocalState
Where YourUsername should be replaced with your Windows user name. Also, the CanonicalGroupLimited.UbuntuonWindows.... part will have a different name depending on the version of Ubuntu that is actually installed.
Referring to the official Microsoft documentation is always the best way to get the most up-to-date and accurate information.
Try it out in practice.
Start powershell or command prompt and type the diskpart command. At this time, I was able to use diskpart without having to open it with administrator privileges, but when I entered the diskpart command, a user account control window appeared. Two windows will be launched; it seems that if you launch powershell with administrator privileges and type in the command, you can use it in one window. The following is displayed.
Microsoft DiskPart version 10.0.22621.1
Copyright (C) Microsoft Corporation.
Computer: N4060
DISKPART>
DISKPART is a Windows command line utility that performs disk partition creation, deletion, formatting, and other disk-related management tasks. File system level operations such as moving, copying, and deleting files and folders are outside the scope of diskpart’s functionality. So enter the following Select the VHD file you are using within diskpart. To do this, use the select vdisk command with the full path of the VHD file. where "C:\path\to\your\vhdx.file" should be replaced with the actual path of the VHD file you want to compress.
select vdisk file="C:\path\to\your\vhdx.file"After the VHD file is selected, enter the following command to perform the compression
compact vdiskThe following is the actual message displayed: ext4.vhdx is in this location since the WSL was duplicated.
DISKPART> select vdisk file="c:\wsl\ubuntu1\ext4.vhdx"
The virtual disk file has been selected by DiskPart.
DISKPART> compact vdisk
100% complete
The virtual disk file was successfully compressed by DiskPart.
DISKPART>
The .vdi file (VirtualBox Disk Image) used by VirtualBox can also be reduced in size to save disk space. VirtualBox does not automatically reduce the size of the disk image as seen by the host OS when a dynamically expanding disk image is used. However, VirtualBox includes a tool to manually reduce the size of the disk image.
Steps to reduce the size of a VirtualBox disk image:
- Shut down the guest OS: Shut down the guest OS. The disk shrink operation cannot be performed while the guest OS is running.
- Fill free space with z eros: Unused disk space in the guest OS must be filled with zeros; for Linux guests, use a command like
sudo dd if=/dev/zero of=/zero.fill; sync; sudo rm -f /zero.fillFor Windows guests, you can use SDelete (a tool provided by Sysinternals) to fill unused space to zero. - Using VirtualBox’s disk management tools: After shutting down the guest OS, open a command line or terminal in the host OS. use VirtualBox’s
VBoxManagecommand to shrink the disk image. The command looks like this (replace the file path and filename as appropriate)VBoxManage modifymedium --compact C:\path\to\your\virtualdisk.vdi
This command compresses the space filled with zeros in the .vdi file, reducing the size of the physical file.
mamu@ubuntu22s:~$ sudo dd if=/dev/zero of=/zero.fill; sync; sudo rm -f /zero.fill
[sudo] password for mamu: $@ubuntu22s:~$ sudo dd: writing to ‘/zero.fill
dd: writing to ‘/zero.fill’: No space left on device
24468585 0 records in
24468584 0 records out
12527915008 bytes (13 GB, 12 GiB) copied, 36.0836 s, 347 MB/s
The command executed and its results are as expected. This command is used to fill unused space in the guest OS with zeros. This process is useful as a preparatory task before reducing the physical size of virtual disks used by virtualization software such as VirtualBox, VMware, or WSL 2.
The parts of the command are explained below.
sudo dd if=/dev/zero of=/zero.fill: writes the data read from/dev/zero(essentially a sequence of zeros) to the/zero.fillfile. This will fill unused space on the guest OS disk with zeros.sync: Forces the file system buffer to be flushed to disk, ensuring that all writes complete to the physical disk.sudo rm -f /zero.fill: deletes used zero-fill files and makes the space available again.
The message dd: writing to ‘/zero.fill’: No space left on device means that all free space on the disk has been filled with zeros and there is no more space left to write to. This is normal behavior and evidence that the purpose of this process has been achieved. The command has written approximately 13 GB of data to the disk and then deleted the files. There is a world of difference between doing this and not doing this.
Following this procedure, an operation to reduce the size of the virtual disk (e.g., the VBoxManage modifymedium --compact command in the case of VirtualBox) can effectively compress the space filled with zeros, resulting in a reduction in the physical size of the virtual disk. This operation is especially useful before backing up or moving disk images.
‘VBoxManage.exe’ is normally found in the following folder
C:\Program Files\Oracle\VirtualBox
Navigate to this folder and run the command (at the command prompt). I got an error if the name of the folder where the virtual hard disk is located contains spaces. If this is the case, move the .vdi file to another folder once. Also, the .vbox file contains a description of the location of the virtual hard disk, and if this file is not moved with the .vbox file, an error will occur. After compression, move the two files to the original location. The error message appears as follows
VBoxManage.exe: error: Cannot register the hard disk ‘C:\temp1\ubuntu22.vdi’ {2bb52558-b25e-49f7-aea3-7280019095e7} because a hard disk ‘C:\ Users\minok\VirtualBox VMs\ubuntu22\ubuntu22.vdi’ with UUID {2bb52558-b25e-49f7-aea3-7280019095e7} already exists
VBoxManage.exe: error: Details: code E_INVALIDARG (0x80070057), component VirtualBoxWrap, interface IVirtualBox, callee IUnknown
VBoxManage.exe: error: Context: “OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, enmAccessMode, fForceNewUuidOnOpen, pMedium. asOutParam())” at line 201 of file VBoxManageDisk.cpp
The reason why deleting files in the guest OS does not increase the capacity at all from the host OS perspective is mainly related to the way disk image files are managed by virtual machines and container technologies (e.g., WSL, VMware, VirtualBox, etc.). Several key factors contribute to this, especially in the case of WSL 2 and virtual machines
1. Dynamically expanding disk images:.
Many virtual machines and WSL 2 create a dynamically expanding type of disk image. This means that the image file occupies only the space it actually uses on disk, and the file size increases as data is added within the guest OS. However, deleting a file in the guest OS does not automatically decrease the size of the disk image file itself. In other words, deleting files does not decrease the actual disk usage.
2. File System Differences:
Guest OSs (especially WSL 2 running Linux, for example) and host OSs (Windows) have different file systems. Deleting a file in the guest OS does not mean that the change is immediately reflected at the host OS file system level. In particular, in WSL 2, the Linux file system is stored in a VHD (virtual hard disk), and the size management of this VHD is independent of the Windows file system.
3. Disk compression and optimization required:
To reduce the physical file size of a dynamic VHD or virtual disk image, certain compression or optimization processes must be performed manually. For example, the Optimize-VHD command (available on Windows Pro and above with Hyper-V enabled) or diskpart’s compact vdisk command can be used to compress unused disk space.
4. Data Fragmentation:.
When a file is deleted in the file system, the space it occupied is marked as “unused,” but the free space on the physical disk is not immediately reorganized. Especially when large numbers of small files are deleted, the actual available free space may appear to be less due to disk fragmentation.
This is why a 300 GB virtual hard disk is now 5 GB.
Zero-embedding and virtual disk compression operations can significantly reduce the physical size of a virtual hard disk. This effect is especially noticeable after much of the data on the disk has been deleted or when there is a large amount of unused space.
Virtualized environments often use dynamically expanding disk image files, and this type of file changes its physical size based on the amount of data actually in use. However, deleting files within the guest OS does not automatically decrease the size of the corresponding disk image file, necessitating a process of zero-filling and compression.
- Zero-filling fills unused space with zeros, allowing the compression process to efficiently identify and compress these areas.
- The compression process removes these zero-filled areas, reducing the physical file size of the virtual disk.
In my case, a 300 GB virtual hard disk was reduced to 5 GB, indicating that this process was very effective. This greatly improves storage utilization, makes backups and moves easier, and saves disk space.