Enhance backup script with pigz support #9

Cristy94 · 2025-11-05T00:44:33Z

Added support for parallel compression and decompression using pigz if available.

Note this is not fully tested.

Related: #8

Added support for parallel compression and decompression using pigz if available.

Cristy94 · 2025-11-05T00:58:03Z

There's an issue with decompressor quotes in remoteCommands (as it's already a quoted string).

Copilot

Pull Request Overview

This PR adds support for using pigz (parallel gzip) for faster compression and decompression during Coolify instance migration. When pigz is available on the system, the script will automatically use it for parallel processing; otherwise, it falls back to standard gzip.

Detects pigz availability and uses it with parallel processing (nproc threads) for compression
Detects pigz availability on the destination server for parallel decompression
Updates tar commands to use the -I flag with custom compressor/decompressor commands

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-05T02:47:21Z

migrate.sh

+  fi
+  
+  # run extraction
+  if ! $decompressor | tar -Px -C /; then


The $decompressor variable is unquoted, which could lead to word splitting and glob expansion issues. Although the variable is set internally in the script, it should be quoted for safety and consistency with the compression command at line 97. Change to if ! "$decompressor" | tar -Px -C /; then.

Suggested change

if ! $decompressor | tar -Px -C /; then

if ! "$decompressor" | tar -Px -C /; then

Copilot · 2025-11-05T02:47:21Z

migrate.sh

+  # Choose compressor
+  if command -v pigz >/dev/null 2>&1; then
+    echo "✅ Using pigz for parallel gzip"
+    compressor="pigz -p$(nproc)"


[nitpick] The -p$(nproc) flag uses all available CPU cores, which could overwhelm the system during migration. Consider limiting the number of threads (e.g., pigz -p$(($(nproc) - 1))) to leave at least one core available for other processes, or make it configurable.

Copilot · 2025-11-05T02:47:21Z

migrate.sh

+    echo 'ℹ️ pigz not found, using gzip';
+    decompressor="gzip -dc"
+  fi
+  


This line contains trailing whitespace. Remove the trailing spaces for consistency with the rest of the script.

Suggested change

…ressing

Cristy94 · 2025-11-08T10:39:12Z

Ok, I think this one should work now, parallel compression and decompression, for 100GB of volumes this results in like 5mins vs 50mins bacup time, a 10x speed-up

Geczy · 2025-11-08T13:16:21Z

did you test it? you said you think lol

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-08T13:19:17Z

migrate.sh

+  # Choose compressor
+  if command -v pigz >/dev/null 2>&1; then
+    echo "✅ Using pigz for parallel gzip"
+    compressor="pigz -p$(nproc)"


The -p$(nproc) option may fail on systems where nproc is not available (e.g., some BSD systems or minimal environments). Consider adding a fallback:

compressor="pigz -p${NPROC:-$(nproc 2>/dev/null || echo 4)}"

Or simply use pigz without the -p flag, as it will automatically use all available cores by default.

Suggested change

compressor="pigz -p$(nproc)"

compressor="pigz -p${NPROC:-$(nproc 2>/dev/null || echo 4)}"

Copilot · 2025-11-08T13:19:17Z

migrate.sh

+      echo '❌ Backup file extraction failed'
+      exit 1


Missing semicolons at the end of statements. The script's existing style (visible in the surrounding remote commands) consistently uses semicolons. Add them for consistency:

else if ! tar -Pzxf - -C /; then echo '❌ Backup file extraction failed'; exit 1; fi fi

Suggested change

echo '❌ Backup file extraction failed'

exit 1

echo '❌ Backup file extraction failed';

exit 1;

Copilot · 2025-11-08T13:19:17Z

migrate.sh

-    exit 1;
+  echo '🚸 Extracting backup file...'
+  if command -v pigz >/dev/null 2>&1; then
+    echo '✅ Using pigz for parallel decompression'


Missing semicolon at the end of the statement on line 173. The script's existing style consistently uses semicolons. Add one for consistency:

echo '✅ Using pigz for parallel decompression';

Suggested change

echo '✅ Using pigz for parallel decompression'

echo '✅ Using pigz for parallel decompression';

Copilot · 2025-11-08T13:19:18Z

migrate.sh

+      echo '❌ Backup file extraction failed'
+      exit 1
+    fi
+  else
+    if ! tar -Pzxf - -C /; then
+      echo '❌ Backup file extraction failed'
+      exit 1


Missing semicolons at the end of statements on lines 175, 176, and 181. While bash doesn't strictly require semicolons before fi, the script's existing style (visible in the surrounding remote commands) consistently uses semicolons. This inconsistency could lead to maintainability issues.

Add semicolons for consistency:

if ! tar -I pigz -Pxf - -C /; then echo '❌ Backup file extraction failed'; exit 1; fi

Suggested change

echo '❌ Backup file extraction failed'

exit 1

fi

else

if ! tar -Pzxf - -C /; then

echo '❌ Backup file extraction failed'

exit 1

echo '❌ Backup file extraction failed';

exit 1;

fi

else

if ! tar -Pzxf - -C /; then

echo '❌ Backup file extraction failed';

exit 1;

Copilot · 2025-11-08T13:19:18Z

migrate.sh

+    fi
  fi
-  echo '✅ Backup file extracted';
+  echo '✅ Backup file extracted'


Missing semicolon at the end of the statement. The script's existing style consistently uses semicolons. Add one for consistency:

echo '✅ Backup file extracted';

Suggested change

echo '✅ Backup file extracted'

echo '✅ Backup file extracted';

Copilot · 2025-11-08T13:19:18Z

migrate.sh

-  if ! tar -Pxzf - -C /; then
-    echo '❌ Backup file extraction failed';
-    exit 1;
+  echo '🚸 Extracting backup file...'


Missing semicolon at the end of the statement on line 171. The script's existing style consistently uses semicolons. Add one for consistency:

echo '🚸 Extracting backup file...';

Suggested change

echo '🚸 Extracting backup file...'

echo '🚸 Extracting backup file...';

Cristy94 · 2025-11-09T20:55:16Z

did you test it? you said you think lol

I tested it, and successfully migrated a few days ago (there were still some issues with some services, where some ENV variables were wrong, or some other small configuration issues, but those are not related to this change).

I said I think this one should work, because I made a few extra small changes after, to fix some warnings.

Maybe the code can be improved, but currently it does seem to workk: use pigz and all threads (maybe the AI suggesiton is good to use threads N-1 instead).

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-29T22:16:45Z

migrate.sh

+  if command -v pigz >/dev/null 2>&1; then
+    echo '✅ Using pigz for parallel decompression'
+    if ! tar -I pigz -Pxf - -C /; then
+      echo '❌ Backup file extraction failed'
+      exit 1
+    fi
+  else
+    if ! tar -Pzxf - -C /; then
+      echo '❌ Backup file extraction failed'
+      exit 1
+    fi
  fi


The extraction logic duplicates error handling code across both branches (pigz and gzip). The conditional could be simplified by determining the decompressor flags first, then calling tar once. This would reduce code duplication and make the logic easier to maintain.

Suggested change

if command -v pigz >/dev/null 2>&1; then

echo '✅ Using pigz for parallel decompression'

if ! tar -I pigz -Pxf - -C /; then

echo '❌ Backup file extraction failed'

exit 1

fi

else

if ! tar -Pzxf - -C /; then

echo '❌ Backup file extraction failed'

exit 1

fi

fi

tar_opts="-Pzxf"

if command -v pigz >/dev/null 2>&1; then

echo '✅ Using pigz for parallel decompression'

tar_opts="-I pigz -Pxf"

fi

if ! tar \$tar_opts - -C /; then

echo '❌ Backup file extraction failed'

exit 1

fi

Copilot · 2025-12-29T22:16:46Z

migrate.sh

+  echo '🚸 Extracting backup file...'
+  if command -v pigz >/dev/null 2>&1; then
+    echo '✅ Using pigz for parallel decompression'
+    if ! tar -I pigz -Pxf - -C /; then


The pigz compression command includes the number of processors with -p$(nproc), but during decompression, pigz is used without the -p flag. While pigz will auto-detect the number of cores by default, it would be more consistent and explicit to also specify the thread count during decompression, especially since it was explicitly set during compression.

Suggested change

if ! tar -I pigz -Pxf - -C /; then

if ! tar -I "pigz -p\$(nproc)" -Pxf - -C /; then

Copilot · 2025-12-29T22:16:46Z

migrate.sh

-  if ! tar --exclude='*.sock' -Pczf $backupFileName -C / $backupSourceDir $HOME/.ssh/authorized_keys $volumePaths; then
+  tar --exclude='*.sock' --warning=no-file-changed -I "$compressor" -Pcf "$backupFileName" \
+    -C / $backupSourceDir $HOME/.ssh/authorized_keys $volumePaths
+  rc=$?


The tar exit code handling allows exit code 1 to pass (which indicates files changed during archival). While this is likely intentional given the --warning=no-file-changed flag, the condition should be "if [ $rc -ne 0 ] && [ $rc -ne 1 ]; then" or "if [ $rc -gt 1 ]; then" to be explicit about which codes are acceptable. The current implementation correctly uses -gt 1, but a comment explaining that exit code 1 is acceptable (files changed during read) would improve clarity.

Suggested change

rc=$?

rc=$?

# tar exit code 0 = success, 1 = files changed during read (acceptable with --warning=no-file-changed)

Enhance backup script with pigz support

6b6b093

Added support for parallel compression and decompression using pigz if available.

Geczy requested a review from Copilot November 5, 2025 02:45

Copilot AI reviewed Nov 5, 2025

View reviewed changes

Fix pigz decompression, and suppress file-changed warnings while comp…

ad90a9b

…ressing

Geczy requested a review from Copilot November 8, 2025 13:16

Copilot AI reviewed Nov 8, 2025

View reviewed changes

Geczy requested a review from Copilot December 29, 2025 22:13

Copilot started reviewing on behalf of Geczy December 29, 2025 22:13 View session

Copilot AI reviewed Dec 29, 2025

View reviewed changes

	if ! $decompressor \| tar -Px -C /; then
	if ! "$decompressor" \| tar -Px -C /; then

	compressor="pigz -p$(nproc)"
	compressor="pigz -p${NPROC:-$(nproc 2>/dev/null \|\| echo 4)}"

	echo '✅ Using pigz for parallel decompression'
	echo '✅ Using pigz for parallel decompression';

	echo '✅ Backup file extracted'
	echo '✅ Backup file extracted';

	echo '🚸 Extracting backup file...'
	echo '🚸 Extracting backup file...';

	if ! tar -I pigz -Pxf - -C /; then
	if ! tar -I "pigz -p\$(nproc)" -Pxf - -C /; then

	rc=$?
	rc=$?
	# tar exit code 0 = success, 1 = files changed during read (acceptable with --warning=no-file-changed)

Enhance backup script with pigz support #9

Are you sure you want to change the base?

Enhance backup script with pigz support #9

Conversation

Cristy94 commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Cristy94 commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Cristy94 commented Nov 8, 2025

Uh oh!

Geczy commented Nov 8, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

Cristy94 commented Nov 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Cristy94 commented Nov 5, 2025 •

edited

Loading

Cristy94 commented Nov 5, 2025 •

edited

Loading