MENU
| Whimsical Blog
  1. Home
  2. education
  3. Convenient use of Rclone (changing the image output destination for stable diffusion)

Convenient use of Rclone (changing the image output destination for stable diffusion)

education
mamushi

Rclone is a command line tool that allows you to efficiently synchronize, backup, and mount data between cloud storage services.

Key Features

  1. Support for many cloud storage services:.
    • Many cloud storage services are supported, including Google Drive, Dropbox, Amazon S3, OneDrive, and more. We actually checked and it supports more than 80 services.
  2. Data Synchronization and Backup:.
    • Easily synchronize and back up data between local storage and cloud storage or between cloud storages.
  3. Mounting Capabilities: The ability to mount
    • Mount cloud storage as if it were a local disk and access it directly from File Explorer.
  4. High performance.
    • High performance for large file transfers and many file operations.

Advantages

  • Open source: Available for free.
  • Versatile: Many options and customization possibilities.
  • Cross-platform: Available for Windows, Mac, and Linux.

We have already shown you how to install it, and now we would like to show you how to apply it, assuming you have installed the Windows and Linux versions. One of the things we are already using is as a backup for WordPress.

Whimsical Blog
What is rclone for cloud storage? | Whimsical Blog A detailed guide on Rclone commands to streamline cloud storage management. Includes extensive explanations and practical examples, making it accessible even fo…

I will show you an example of using Rclone for Windows and Linux versions to make Stable Diffusion’s image output destination an online storage. The images will be output to the following location

C:\youtube\stable-diffusion-webui\outputs

I used to use symbolic links to achieve this, but when I tried it with Rclone, it was more convenient than I expected. I tried it with Rclone and found it more convenient than I expected, since it also supports sftp, so it can be used effectively by storing it on Lollipop’s extra storage space, for example. Now, let’s get to the actual work.

Assuming that Rclone is already installed and remote configuration has been completed, configure remote storage (e.g. Dropbox, Google Drive, etc.) with the rclone config command. The remote configuration is SFTP, which is commonly found on Japanese rental servers (I set it up with a Lollipop account). First, mount the remote storage to the folder where the images will be output.

rclone mount losftp:web/stable C:\youtube\stable-diffusion-webui\outputs

It is safe to delete files and folders under outputs. In other words, the contents of outputs should be empty. If necessary, backup the files and folders before deleting them. However, I got an error.

Fatal error: failed to mount FUSE fs: mountpoint path already exists: C:\youtube\stable-diffusion-webui\outputs

To use Rclone’s mount function on Windows, you may need WinFsp (Windows File System Proxy). So download and install it from the official website.

https://winfsp.dev

WinFsp (Windows File System Proxy) is a toolkit for implementing a user-space file system in a Windows environment. With this tool, developers can implement custom file systems in user-space and make them accessible through the standard Windows file system interface.

WinFsp Roles and Functions

  1. Userspace file system implementation:.
    • While filesystems are normally implemented in kernel mode, WinFsp allows users to implement filesystems in userspace. This facilitates development and debugging.
  2. Support for custom filesystems:.
    • Developers can create custom file systems to suit their specific needs. For example, cloud storage or remote file systems can be treated as local file systems.
  3. FUSE Compatibility:.
    • WinFsp is compatible with Linux’s Filesystem in Userspace (FUSE), which is useful for porting FUSE-based file systems to Windows.

WinFsp and Linux File System Problem Solving

WinFsp itself is not a tool that directly solves compatibility issues between Windows and Linux file systems, but it can be useful in the following ways

  1. FUSE file system porting:.
    • WinFsp can be used to port a Linux FUSE-based file system to Windows so that it can be accessed by Windows applications. This allows cross-platform file system access.
  2. Creating a custom file system:.
    • Developers can use WinFsp to create custom filesystems that are compatible with specific Linux filesystems (e.g., ext4, XFS) and make them accessible from Windows.
  3. Cloud and remote storage integration:.
    • Combined with a tool like Rclone, cloud or remote storage can be mounted as a Windows file system and accessed seamlessly.

Run the downloaded installer and follow the on-screen instructions. Make sure it is running as a service after installation. Open the Services Manager ( services.msc ) and make sure “WinFsp.Launcher” is “running”. Or you can also do the following WinFsp is usually installed in C:\Program Files (x86)\WinFsp. Run fsptool-x64.exe here to verify that WinFsp is working properly.

Enter the command to mount again and this time it works. I later realized that I also needed to delete the outputs itself; without WinFsp installed, I would get an error even without the outputs folder.

When mounting remote storage with Rclone, the mounted folder icon may be different from the normal folder icon. This indicates that the mounted remote storage is using a particular file system or protocol.

Why are the icons different?

  1. File system differences:.
    • Mounted remote storage uses a different file system than the local file system and may display different icons.
  2. Mount Point Identification: The
    • The operating system may use a different icon to distinguish a mount point from a regular folder. This allows users to recognize at a glance that the folder is remote storage.

Examples

  • On Windows:.
    • A mounted remote storage folder may have an icon that looks like a network drive.
  • For Linux: The
    • Mount point folders may be marked differently than normal folder icons.

Now you can automatically configure the mount settings after rebooting. To achieve this, we initially created a batch file and ran it at system startup using the Windows task scheduler, but the command prompt kept coming up, which was uncomfortable. So we decided to use NSSM to run Rclone as a Windows service.

https://nssm.cc

How to use NSSM (Non-Sucking Service Manager)

When Rclone is run as a Windows Service using NSSM, the command prompt window does not appear and it runs in the background as a service, so closing the window does not dismount it.

  1. Download and install NSSM
    • Download NSSM from the official NSSM website.
    • Unzip the downloaded ZIP file and place nssm.exe in an appropriate location (e.g. C:\nssm\nssm.exe ).
  2. Install Rclone as a service using NSSM
    • Open a command prompt as an administrator and execute the following command
    C:\nssm\nssm.exe install rclone_mount
  3. Configure the service
    • The NSSM interface will open and you can configure the following settings
    Application tab:.
    • Specify the full path of rclone.exe in Path (e.g. C:\Program Files\rclone\rclone.exe ) Specify the directory path of Rclone in Startup directory Enter the following in Arguments: mount losftp:web/stable C:\youtube\stable-diffusion-webui\outputs --vfs-cache-mode writes --config "C:\Users\yourusername.config\rclone\ rclone.conf"
    Details tab:.
    • Enter a descriptive service name in Display name (e.g. Rclone Mount ).
  4. Start the service
    • Save the NSSM configuration and start the service with the following command.
    net start rclone_mount
    Or start it with Windows Service Manager. If it is even slightly wrong, an error will occur.

--config About "C:\Users\yourusername.config\rclone\rclone.conf
If you look at rclone.conf with Notepad, you will find the authentication information. rclone.conf location can be checked with the following command.

rclone config file

–vfs-cache-mode About writes
How files in online storage mounted by Rclone are handled depends on the VFS cache mode used. rclone has several VFS cache modes, each of which handles files in a different way. Below is an overview of the main modes

  1. Off (off):
    • Default mode.
    • Files are read/written directly from/to remote storage.
    • They are treated as online files and downloaded on demand.
    • No temporary cache is used.
  2. Read cache(MINIMAL):
    • Data is temporarily cached while the file is being read.
    • It is treated as an online file and downloaded as needed.
    • The cache is cleared when the file is read.
  3. Writecache(writes):
    • Files are cached locally and uploaded to the remote after the write operation is complete.
    • When reading, data is retrieved directly from the remote.
    • Primarily used to improve write performance.
  4. Full cache(full ):
    • Both reads and writes are cached locally.
    • The file is downloaded on first access and then accessed from the local cache.
    • It is treated like an offline file and is kept locally until the cache is deleted.

Ultimately, regardless of the mode, the file will eventually be saved to online storage. Rclone will now run in the background as a Windows service. Use the following command to remove the service

nssm remove service name

Another possible alternative to Rclone is to use a script that moves files once they have been saved locally. For example, a batch file or shell script could be used to periodically check for files stored in a local directory and move them to remote storage. This makes it easy to support specific cloud storage services.

Script Examples

Windows batch file

@echo off 
setlocal 

REM local directory 
set LOCAL_DIR=C:\path\to\local\directory 

REM remote directory 
set REMOTE_DIR=\path\to\remote\directory 

REM local Move files in a directory to a remote directory 
move "%LOCAL_DIR%\*" "%REMOTE_DIR%" 

endlocal

Linux shell scripts.

#! /bin/bash 

# local directory 
LOCAL_DIR="/path/to/local/directory" 

# remote directory 
REMOTE_DIR="/path/to/remote/directory" 

# move files in local directory to remote directory 
mv "$LOCAL_DIR"/* "$REMOTE_DIR"

These scripts can be used to periodically move files stored in a local directory to a remote directory. This method can also be applied to certain cloud storage services and remote servers.

For real-time performance, a more effective method is to monitor file system events and move files the moment they are created. Below is a description of how to move files in real-time.

Windows Environment

In Windows, PowerShell can be used to monitor file system events and create scripts to move files in real time when they are created.

Example PowerShell script

$watcher = New-Object System.IO.FileSystemWatcher 
$watcher.Path = "C:\path\to\local\directory" 
$watcher.Filter = "*.*" 
$watcher.IncludeSubdirectories = $false 
$watcher.EnableRaisingEvents = $true 

$action = { 
 $sourcePath = $Event.SourceEventArgs.FullPath 
 $ destinationPath = "C:\path\to\remote\directory\" $Event.SourceEventArgs.Name 
 Move-Item -Path $sourcePath -Destination $destinationPath 
} 

Register-ObjectEvent $watcher "Created" -Action $action 

# Wait without exiting script 
while ($true) { Start-Sleep 1 }

Linux Environment

In Linux, the inotify tool can be used to create scripts that monitor file system events and move files in real time when they are created.

Example shell script.

#! /bin/bash 

LOCAL_DIR="/path/to/local/directory" 
REMOTE_DIR="/path/to/remote/directory" 

inotifywait -m -e create "$LOCAL_DIR" | 
while read path action file; do 
 mv "$LOCAL_DIR/$file" "$REMOTE_DIR" 
done

macOS environment

Under macOS, the fswatch tool can be used to monitor file system events and create scripts that move files in real time as they are created.

Example shell script.

#! /bin/bash 

LOCAL_DIR="/path/to/local/directory" 
REMOTE_DIR="/path/to/remote/directory" 

fswatch -0 "$LOCAL_DIR" | while read -d "" event; do 
 mv "$ event" "$REMOTE_DIR" 
done

The next step is to set up something similar for ubuntu: on Windows, I got an error when trying to mount against an existing folder, but on Ubuntu, I was able to mount Rclone against an existing directory with no problem. The following commands mount manually

rclone mount dropb:stableout /home/mamu/stable-diffusion-webui/outputs

Having succeeded manually, there are instructions on the official page on how to run it as a systemd service. This is the reference to set it up.

https://rclone.org/commands/rclone_mount

Do the following

Servicing Rclone is necessary to automatically mount remote storage, even after a system reboot. Symbolic links and unit file settings play an important role in accomplishing this. Below is an explanation of why each of these elements is necessary.

1. Symbolic Links

What is a symbolic link?

A symbolic link is a special file that points to a reference to a certain file or directory. Think of it like a shortcut.

Why do we need symbolic links?

When using Rclone to mount remote storage as a file system, a special file path is required for some tools on the system to recognize Rclone. By creating a symbolic link, Rclone will be recognized as that special file path and will work smoothly. Specifically, the system recognizes Rclone as a “mount tool” and handles it appropriately.

2. Unit File

What is a unit file?

A unit file is a configuration file that allows the system to manage specific services and tasks. Creating it lets the system know how and when to start a particular service.

Why do we need a unit file?

A unit file is like a set of instructions for automating the mounting of remote storage. two unit files are needed: a unit file for the remote storage system, and a unit file for the remote storage system.

  1. Mount unit file
    • This defines how the remote storage is to be mounted. It specifies the location of the remote storage (for example, a specific folder in Dropbox) and the mount point (where to mount it).
  2. Automount Unit File
    • This tells the system to automatically initiate a mount when it is needed. For example, it will automatically initiate a mount when the system is rebooted or when a specific directory is accessed.

Conclusion

  • Symbolic links are “shortcuts” that allow Rclone to be recognized by some of the system’s tools.
  • The unit file is a “configuration file” that tells the system how and when to automatically mount remote storage.

By setting it up this way, the system will automatically mount the remote storage even after a reboot, eliminating the need for the user to perform the mount operation manually. First, create a symbolic link.

sudo ln -s /usr/bin/rclone /sbin/mount.rclone

Next, create the mount unit file. The description and file name can be rewritten to suit your environment, referring to the contents of the official Rclone page. Actually, this file name is wrong, as I will write later.

sudo nano /etc/systemd/system/home-mamu-stable-diffusion-webui-outputs.mount
[Unit].
Description=Mount for /home/mamu/stable-diffusion-webui/outputs
After=network-online.target

[Mount] Description=Mount for /home/mamu-stable-diffusion-webui/outputs
What=dropb:stableout
Where=/home/mamu/stable-diffusion-webui/outputs
Type=rclone
Options=rw,_netdev,allow_other,args2env,vfs-cache-mode=writes,config=/home/mamu/.config/rclone/rclone.conf,cache-dir=/var/rclone

[Install]
WantedBy=multi-user.target

Matching unit file names and directory paths

Why do unit file names and directory paths need to match?

When managing unit files in a system, it is important that the file names and their contents are consistent. This is so that the system will automatically find and apply the correct unit file. Matching unit file names and directory paths makes it easier for the system to identify the correct file and make it work as intended.

Specific reasons

  1. Ensure consistency
    • Matching unit file names and directory paths makes it clear which unit file corresponds to which mount point. This facilitates administration and prevents misunderstandings and errors.
  2. Automatic Recognition
    • The system automatically recognizes the units to be applied using the unit file name. For example, a unit file named home-mamu-stable-diffusion-webui-outputs.mount will recognize the /home/mamu/stable-diffusion-webui/outputs directory as a mount point The unit file named
  3. Ease of Debugging
    • If a problem occurs, matching unit file names and directory paths can quickly identify which files are at fault. This facilitates troubleshooting.

Explanation with concrete examples

Example 1: Unit file name and directory path match

  • Unit file name: home-mamu-stable-diffusion-webui-outputs.mount
  • Directory path: /home/mamu/stable-diffusion-webui/outputs

In this case, the system automatically recognizes that this unit file is for mounting the /home/mamu/stable-diffusion-webui/outputs directory.

Example 2: Unit file name and directory path do not match

  • Unit file name: example.mount
  • Directory path: /home/mamu/stable-diffusion-webui/outputs

In this case, the system cannot determine exactly which directory should be mounted, resulting in misconfiguration and malfunction. Yet another automount unit file is created. The name of this file is also incorrect and will be explained later.

sudo nano /etc/systemd/system/home-mamu-stable-diffusion-webui-outputs.automount
[Unit].
Description=AutoMount for /home/mamu/stable-diffusion-webui/outputs

[Automount] Where=/home/mamu/stable-diffusion-webui-outputs.automount
Where=/home/mamu/stable-diffusion-webui/outputs
TimeoutIdleSec=600

[Install]
WantedBy=multi-user.target

When you are ready, reload the systemd daemon to reflect the changes in the unit file.

sudo systemctl daemon-reload

Normally it is sufficient to enable the automount unit, but there may be times when it is necessary to manually enable and start the mount unit. Enable and start the automount unit.

sudo systemctl enable home-mamu-stable-diffusion-webui-outputs.automount
sudo systemctl start home-mamu-stable-diffusion-webui-outputs.automount

However, an error occurred in the startup command.

Failed to start home-mamu-stable-diffusion-webui-outputs.automount: Unit home-mamu-stable-diffusion-webui-outputs.automount has a bad unit automount has a bad unit file setting.
See system logs and ‘systemctl status home-mamu-stable-diffusion-webui-outputs.automount’ for details.

There is a description “cache-dir=/var/rclone” in the mount unit file. However, upon examination, it is not there, so create it and set the appropriate permissions.

sudo mkdir -p /var/rclone
sudo chown mamu:mamu /var/rclone

There is a further problem with the escape sequence, and the above alone did not resolve the error.

What is an escape sequence?

An escape sequence is a special code that allows certain characters and symbols to be represented correctly. They are used to make certain environments and programs interpret characters and symbols correctly.

Why do we need escape sequences?

On Linux systems, certain characters such as slashes ( / ) and hyphens ( - ) must be converted to escape sequences when using directory paths in unit file names. Understanding this will help the system understand and properly handle the correct path.

Specific examples of escape sequences

Escaping Hyphens

When a hyphen (-) is used in a unit filename, it must be converted to an escape sequence. Specifically, the hyphen (-) is represented as \\x2d in the escape sequence.

Example: Conversion of directory paths and unit file names

For example, if the directory path is /home/mamu/stable-diffusion-webui/outputs, when converting this path to a unit file name, the hyphen is converted to the escape sequence

  • Directory path: /home/mamu/stable-diffusion-webui/outputs
  • Unit file name: home-mamu-stable\\x2ddiffusion\\x2dwebui-outputs.mount

Differences with Windows

In Windows, the concept of escape sequences is not very common; the Windows file system has a different concept of unit files and mount points, so escape sequences are often not needed. However, Linux requires the use of escape sequences when including certain characters in file paths and unit file names.

Conclusion

Escape sequences are special codes that accurately represent certain characters and symbols; when converting a unit file name to a directory path on a Linux system, certain characters such as slashes and hyphens must be converted to escape sequences. This will allow the system to understand the correct path and handle it properly.

In this way, matching the unit file name with the directory path allows the system to correctly recognize and apply the unit file. Taking this into account, the following is the correct answer.

The file names will be as follows, respectively

home-mamu-stable\x2ddiffusion\x2dwebui-outputs.mount
home-mamu-stable\x2ddiffusion\x2dwebui-outputs.automount

Following this, the service is also enabled and started as follows

sudo systemctl enable home-mamu-stable\\x2ddiffusion\\x2dwebui-outputs.automount
sudo systemctl start home-mamu-stable\\x2ddiffusion\\x2dwebui-outputs.automount

Escape sequences may be needed in a variety of situations outside of unit files. Below are some common cases where escape sequences should be noted.

General Cases

1. Shell scripts and command lines

Sometimes it is necessary to escape certain characters in shell scripts or on the command line. For example, special characters such as spaces, asterisks ( * ), and dollar signs ( $ ) require escaping.

Example: A directory containing spaces

mkdir "My Folder" 
cd My Folder

2. Regular Expressions

Regular expressions can use metacharacters ( ., *,?, [], etc.) must be escaped.

Example: treat dots as literals

grep "example\.com" file.txt

3. Programming Languages

Programming languages (Python, JavaScript, etc.) also require escaping special characters in strings.

Example: Escaping in Python

path = "C:\\Users\\name\\Documents"

4. URL Encoding

Certain characters must be escaped in URLs. For example, spaces are encoded in %20.

Example: URL encoding

https://www.example.com/search?q=hello%20world

5. SQL Queries

It is necessary to escape special characters in SQL queries. This is especially the case for string literals containing single quotes ( ' ).

Example: Escaping in an SQL query

SELECT * FROM users WHERE name = 'O''Reilly';

Examples of using escape sequences outside of unit files

1. Example in a shell script.

#! /bin/bash 
echo "This is a backslash: \\" 
echo "This is a new line character: \n"

2. Example with regular expression (Bash)

echo "The quick brown fox" | grep "quickɑ brown"

3. Example in Python

# Escaping file paths 
path = "C:\\Users\\name\\Documents\\file.txt"

# Escaping special characters 
text = "This is a double quote: \" and this is a single quote: \'"

Conclusion.

Escape sequences are needed to treat certain characters literally or to change the meaning of certain characters. You need to use escape sequences in a variety of situations, including shell scripts, regular expressions, programming languages, URL encoding, and SQL queries. It ensures that code and commands work as expected and prevents errors and unexpected behavior.

education
Please share if you like it!
TOC