MENU
| Whimsical Blog
  1. Home
  2. education
  3. Other disks were used because of the large capacity of the image generation AI model.

Other disks were used because of the large capacity of the image generation AI model.

education
mamushi

Because of the extremely large size of stable diffusion models, you can place the models on USB or another PC so that they do not reference their normal location. By doing this, you will be able to use the model on multiple computers. Alternatively, you can also share models when you have multiple installations of the stable diffusion environment on a single PC. This is recommended because duplicating the same model wastes disk space.

It is conceivable to use techniques such as virtual folders or shortcuts to circumvent the limitations of software that does not allow changing the location of data storage. The following methods are possible

  1. Symbolic Links:
    • Symbolic links can make a file or directory appear to be in a different location. This allows the software to think it is storing data in a specific location, when in fact the data is stored in a different location.
  2. Mount Point:
    • A specific point in the file system can be mounted on another disk or directory to change the location of data storage.
  3. Virtual Disk.
    • Create a virtual disk and treat it like a real disk to control where data is stored.
  4. Data Redirection.
    • Some software and services support data redirection, which allows data to be redirected to another location for reading or writing.
  5. File System Filter Driver:
    • File system filter drivers can modify file system behavior and control where data is stored.

These methods require technical knowledge and may affect your system and data, so it is important to exercise caution. It is also important to be careful that these methods do not violate software license terms or terms of use.

In this article, we will use symbolic links to solve the problem.

A symbolic link (Symbolic Link, or symlink) is a special file that is a reference to another file or directory on the file system. A symbolic link can be thought of as a shortcut to a file or directory. A symbolic link maintains the path of the original file or directory and redirects to that path when accessed.

Here are some key points:

  1. Referentiality:
    • Symbolic links serve as pointers to the target file or directory. The symbolic link itself does not hold the contents of the target, but it does hold the path to the target.
  2. Relative and absolute paths:
    • Symbolic links can hold either relative or absolute paths to the target.
  3. Lightweight:
    • Symbolic links consume very little space on disk because they contain no actual data.
  4. Portability:
    • Symbolic links can be created in a different directory or device than the target.
  5. Creation and deletion:
    • When a symbolic link is created or deleted, the target file or directory is not affected.

Symbolic links are especially useful for managing data in large file systems and for flexible program configuration. Symbolic links are also widely used in UNIX and UNIX-like operating systems (e.g., Linux and macOS). symbolic links can be created in Windows, but they are a different concept from “shortcuts” and are only supported on NTFS file system.

We will practice this in the Windows and Linux environments. First, we will set up in the Windows environment. as an example of how to circumvent software restrictions on storing data in a particular folder in the Windows environment, we will discuss the use of symbolic links. This method may be useful because symbolic links allow you to link one directory to another. First, open a command prompt as an administrator. At the command prompt, type the following command and press enter

mklink /D "linked path" "actual data path"

For example, assuming that the software stores data in C:\ProgramData\SoftwareData, and you want to store the actual data in D:\SharedData, enter the following command A word of caution here. If an existing directory exists at the location where you intend to create the symbolic link, you must first rename or move the existing directory. This will ensure an “empty location” for the symbolic link to be created.

mklink /D "C:\ProgramData\SoftwareData" "D:\SharedData"

Now the software will assume that the data is stored in C:\ProgramData\SoftwareData, when in fact the data is stored in D:\SharedData.

CAUTION:

  • Before creating a symbolic link, make sure that the C:\ProgramData\SoftwareData folder does not exist or contain critical data. Move or delete folders as necessary. Example (temporarily name the folder model as model1, etc.)
  • Symbolic links provide a way to redirect software to another location without changing the location of files and directories, but this may cause unexpected behavior or problems. Therefore, it is strongly recommended that you make a backup of your critical data before performing this method.

The command entered in the actual environment is shown below.

mklink /D "C:\youtube\stable-diffusion-webui\models\Stable-diffusion" "F:\model"

The F drive is USB, but a network drive is also possible. However, it takes a long time to load models. It may be good for use with multiple PCs. It could actually generate images.

Next we will check it in Ubuntu Desktop environment.

To create a symbolic link in the actual Ubuntu environment, use the ln command. Please follow the steps below.

Open a terminal:

  • Open a terminal on your Ubuntu desktop.

Create a symbolic link: In the terminal, run the following command to create a symbolic link.

  • In the terminal, execute the following command to create a symbolic link
ln -s <path_to_actual_data> <path_to_symlink>

For example, if the actual data is located in /mnt/d/SharedData and you want to link it to /home/<your-username>/SoftwareData, execute the following command

ln -s /mnt/d/SharedData /home/<your-username>/SoftwareData

To check:

  • To verify that the symbolic link has been created correctly, execute the following command
ls -l <path_to_symlink>

Notes:

  • When accessing Windows data on Ubuntu, it is usually through the drive letter under /mnt/. For example, the D: drive is accessed as /mnt/d. However, this works in a WSL environment, not in a normal dual-boot or virtual machine environment. To access Windows data in a real Ubuntu environment, network shares or other methods must be used.
  • Symbolic links assume that the referenced file or directory exists, so please make sure that the referenced path is correct before creating the link.
  • If an error occurs when creating a symbolic link, check for permission issues. If necessary, use the sudo command to create the symbolic link.

The actual command entered is shown below, the data is on the USB.

ls -s /media/yourname/USB volume name/model /home/mamushi/stable-diffusion-webui-docker/models/Stable-diffusion

Normally the above method is possible, but since it was the Docker version of stable diffusion, I could not access the models.

I used the following GitHub for building in Docker.

https://github.com/AbdBarho/stable-diffusion-webui-docker

The following was important

Symbolic links are not resolved correctly within a Docker container because the target of the symbolic link resides on the host machine outside the container The Docker container has its own file system, isolated from the host machine’s file system. This means that files and directories on the host machine cannot be accessed directly from within the container.

Creating a directory on the host machine after the fact does not automatically make it available in the Docker container. To access a directory on the host machine from within a container, you must explicitly mount that directory in the container using Docker’s mount functionality. This can be done by using the -v or --volume option of the docker run command or by editing the volumes section of the docker-compose.yml file.

So, I made a modification to docker-compose.yml. After analyzing the container, I found that the /data/models/Stable-diffusion directory was the location of the models.

x-base_service: &base_service
    ...
    volumes:
      - &v1 ./data:/data
      - &v2 ./output:/output
      - /media/mamushi/UBUNTU-SERV/model:/data/models/Stable-diffusion  # lines to add
    ...

Mount the default location on a USB drive to load the models.

education
Please share if you like it!
TOC