Roadmap Discussion: Future Direction of cloudberry-backup

[tuhaihe]

Background

As the cloudberry-backup project continues to evolve, we want to ensure that our development priorities align with the community's needs. We've recently integrated gpbackman as our open-source backup management tool and are now planning the next phase of development.

Current Status

cloudberry-backup currently provides:

• gpbackup and gprestore utilities for backup and restore
• Incremental backup support
• Plugin support (S3, etc.)
• gpbackman integration for backup lifecycle management

We Need Your Input!

We'd love to hear from the community:

1. Missing Features: Are there any critical features we've missed that you need for your backup workflows?

2. Pain Points: What are the biggest challenges you face with the current backup tools?

Please share your thoughts by:

• Commenting on this issue with your feedback
• Sharing your specific use cases or requirements
• Suggesting additional features or improvements

Related Resources

• gpbackman GitHub Repository
• VMware gpbackup-manager Documentation

---

Thank you for being part of the cloudberry-backup project! Your feedback will directly shape the future of this project. 🙏

[woblerr]

Hi

1. Monitoring: Prometheus Metrics Exporter

There is currently no built-in way to monitor backup status via metrics. I have a ready-to-use Prometheus exporter — gpbackup_exporter — that collects metrics from the gpbackup_history.db file. It already uses some calls from the original gpbackman library (which is now integrated into cloudberry-backup).

The exporter provides the following metrics:

Metric	Description
gpbackup_backup_status	Success/failure per backup
gpbackup_backup_deletion_status	Deletion lifecycle tracking
gpbackup_backup_info	Backup metadata (version, compression, plugin, etc.)
gpbackup_backup_duration_seconds	Backup duration
gpbackup_backup_since_last_completion_seconds	Time since last completed backup
gpbackup_exporter_status	Exporter health / data fetch status

There is also a ready-made Grafana dashboard for visualization.

If this is relevant to the project's roadmap, I can port the exporter into cloudberry-backup.

2. Backup Encryption

Currently backup data is stored unencrypted. All modern backup utilities (including WAL-G for physical backups) support data encryption.

I would suggest splitting this feature request into two parts:

• S3 plugin encryption (higher priority): Encrypt backup data before uploading to S3. This is critical because object storage is often shared or accessible over the network, making unencrypted data a significant security risk.

• Local/mounted storage encryption: Encrypt backups when data is stored on local disks or mounted file storage (i.e., when --backup-dir is used with gpbackup). While the risk is somewhat lower than with S3 (since access control is handled at the OS/filesystem level), encryption at rest is still a best practice.

Of course, encryption should be an option, and users can choose whether to use it or not.

In addition, it would be valuable to provide documentation on how to implement encryption when using custom plugins, so that third-party plugin authors can follow a consistent approach.

[gspiegelberg]

sqlite database is fine however one point of backups is to get a copy off the cluster & host in case of disaster. This should include the database behind the DR solution.

Suggest adding alternative, such as an off-host/remote PostgreSQL database, where gpBackMan information is stored. Information is then more available and negates the necessity for gpbackman_exporter and Prometheus which not every organization has or wants.

gpbackman --host=db.host.local --port=5432 --username=gpbackman_role --dbname=gpbackman ...

Connection options:
  -d, --dbname=DBNAME      database name to connect to
  -h, --host=HOSTNAME      database server host or socket directory
  -p, --port=PORT          database server port
  -U, --username=USERNAME  database user name
  -w, --no-password        never prompt for password
  -W, --password           force password prompt (should happen automatically)

[leborchuk]

Add wal-g support.

wal-g has their own metadata format, but the limited set of commands to view backup and their metrics.

We could add support not only logical but also physical backups to the backup management utility.

So, users can have the same interface for working with all kinds of backups in their cluster. The closest analog is Oracle Recovery Manager (RMAN), which started as a simple utility, but has now become a major citizen in the Oracle ecosystem https://www.oracle.com/database/technologies/high-availability/rman.html

[woblerr]

cloudberry-backup currently uses go 1.21, which is already eol. At the same time, cloudberry-go-libs has already been bumped to go 1.24.

It would be great to bump go version to 1.24. It also makes sense to coordinate this with cloudberry-pxf (also on go 1.21) to keep consistency across the ecosystem.

[tuhaihe]

cloudberry-backup currently uses go 1.21, which is already eol. At the same time, cloudberry-go-libs has already been bumped to go 1.24.

It would be great to bump go version to 1.24. It also makes sense to coordinate this with cloudberry-pxf (also on go 1.21) to keep consistency across the ecosystem.

Hi @woblerr, thanks for raising this important point. Could you help create an Issue in the cloudberry-pxf repo to initiate a discussion about upgrading its Go version to 1.24. This will allow for a coordinated approach to these updates. Thanks again!

Update: apache/cloudberry-pxf#96