* Replace pigz with zstd for backup compression
This change replaces pigz (parallel gzip) with zstd (Zstandard) as the
compression algorithm for mailcow backups while maintaining full backward
compatibility with existing .tar.gz backups.
Benefits:
- Better compression ratios (12-37% improvement in tests)
- Improved compression speed with modern algorithm
- Maintains rsyncable functionality for incremental backups
- Full backward compatibility for restoring old .tar.gz backups
- Wide industry adoption and active development
Changes:
- Backup compression: pigz --rsyncable -p → zstd --rsyncable -T
- Backup decompression: pigz -d -p → zstd -d -T
- File extensions: .tar.gz → .tar.zst
- Added get_archive_info() function for intelligent format detection
- Updated backup Dockerfile to install zstd alongside pigz
- Restore function now auto-detects and handles both formats
- Updated FILE_SELECTION regex to recognize both .tar.zst and .tar.gz
- Updated comments to reflect new file extension
Backward Compatibility:
- Restore automatically detects .tar.zst (preferred) or .tar.gz (legacy)
- Existing .tar.gz backups can still be restored without issues
- pigz remains installed in backup image for legacy support
- Graceful fallback if backup file format not found
Testing:
- Added comprehensive test suite (test_backup_and_restore.sh)
- 12 automated tests covering all scenarios:
* Backup creation (both formats)
* Restore (both formats)
* Format detection and priority
* Error handling (missing files, empty dirs)
* Content integrity verification
* Multi-threading configuration
* Large file compression (8.59 MB realistic data)
Test Results:
✓ zstd compression working
✓ pigz compression working (legacy)
✓ zstd decompression working
✓ pigz decompression working (backward compatible)
✓ Archive detection working
✓ Content integrity verified
✓ Format priority correct (.tar.zst preferred)
✓ Error handling for missing files
✓ Error handling for empty directories
✓ Multi-threading configuration verified
✓ Large file compression: 37.05% improvement
✓ Small file compression: 12.18% improvement
* move testing script into development folder
---------
Co-authored-by: DerLinkman <niklas.meyer@servercow.de>
This commit fixes two bugs in the cold-standby script:
1. Support digits in COMPOSE_PROJECT_NAME
The script was stripping digits from COMPOSE_PROJECT_NAME, while
backup_and_restore.sh (fixed in a71d991c) correctly supports them.
Added '0-9' to the tr character set to align behavior.
2. Support docker-compose.override.yml on remote
Lines 172 and 287 explicitly used '-f docker-compose.yml' which
causes Docker Compose to ignore docker-compose.override.yml even
when present. Changed to 'cd && compose' pattern (matching line 296)
to auto-discover override files.
Impact: Users with custom volumes/services in override file would
experience silent failures - volumes not created, images not pulled,
data syncing to wrong locations.
Both fixes ensure cold-standby works correctly with standard Docker
Compose conventions and user customizations.
Co-authored-by: Christophe Neuerburg <c.neuerburg@sdsys.ch>
* fts-flatcurve: inital implementation
* fts: removed solr from compose.yml
* flatcurve: added heap and proc logic to dovecot
* added logic for update.sh & generate for Flatcurve
* delete old iteration of fts-flatcurve.conf
* updated default fts.conf
* updated .gitignore to exclude fts.conf for further git updates
* Remove autogeneration of fts.conf (disable override)
* cleanup all left solr stuff
* renamed SKIP_FLATCURVE to SKIP_FTS
* cleanup leftovers solr in lang files
* moved lazy_expunge plugin only to mail_plugins
* added fts timeout value
* compose: remove dev image of dovecot
* updated japanese translation
This is related to https://github.com/mailcow/mailcow-dockerized/issues/5970 and https://community.mailcow.email/d/2126-backup-restore/2
It adds `docker compose create` to the script which gets executed directly after the sync of the mailcow-dockerized directory. This way the Docker daemon on the remote side creates everything and we get rid of the warning "volume "XYZ" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume"
This is helpful if you use the create-cold-standby.sh script to migrate your mailcow installation to another server and don't want to get those warnings after migration.
Co-authored-by: Niklas Meyer <niklas.meyer@servercow.de>
