Merged
Conversation
9c69644 to
352bea8
Compare
Add option to set up a pgbouncer server that can manage traffic to the actual database BREAKING CHANGE: The `db` argument in the TiPgApiLambda, PgStacApiLambda, and TitilerPgstacApiLambda constructs has been replaced with `connectionTarget`. This object is used to add the Lambdas to the list of acceptable connections and can be either an RDS Instance or an EC2 instance running pgbouncer. also install new pgbouncer-secret-updater package in ci
Contributor
Author
|
@vincentsarago I have it set up to work with Lambdas in or out of the VPC. Now the I added some more notes about networking setups over here: #116. |
vincentsarago
approved these changes
Jan 22, 2025
github-actions bot
pushed a commit
that referenced
this pull request
Jan 24, 2025
# [7.4.0](v7.3.0...v7.4.0) (2025-01-24) ### Features * add pgbouncer ([#114](#114)) ([5952858](5952858))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add option to set up a
PgBouncerserver that can efficiently route traffic to the actual databaseMerge request description
Add option to set up a pgbouncer server that can manage traffic to the actual database.
Adding a connection pooler like PgBouncer to eoAPI drastically improves performance by preventing the database instance from being overwhelmed by connections from eager clients like titiler-pgstac. AWS offers the RDS Proxy as a managed version of this setup, but it is quite expensive and can double the cost of running an RDS instance. By setting up an EC2 instance running PgBouncer, we can get the same impact for a fraction of the RDS Proxy cost (<$10/mo).
Most of the work to get PgBouncer running was done in MAAP-Project/maap-eoapi#39. This PR builds off of that work and does a more complete job of integrating the PgBouncer server with the rest of the eoAPI applications (not just titiler-pgstac).
I did that by creating a new secret that contains the same values as the original PgstacSecret but with the PgBouncer EC2 instance private or public IP address (depending on the networking setup) substituted for the
hostkey. If the user specifiesaddPgBouncer: truein thePgStacDatabaseconstruct, an EC2 instance running PgBouncer will be created and thepgstac_secretattribute will be set to the PgBouncer secret instead of the original PgStac secret. This makes it easy to configure client apps to connect to the database via PgBouncer instead of making a direct connection to the database.The PgBouncer infrastructure is slightly more complicated than I want it to be because we cannot create the PgBouncerSecret until the EC2 instance has been created and assigned an IP address. To get around this, I set up a Lambda function to create the PgBouncer secret with nothing in it, then update it with the correct values after the EC2 instance is running.
Breaking changeThedbargument in the TiPgApiLambda, PgStacApiLambda, and TitilerPgstacApiLambda constructs has been replaced withconnectionTarget. This object was/is used to add the Lambdas to the list of acceptable connections and now it can be either an RDS Instance or an EC2 instance (running PgBouncer). I made the breaking change because it would be very easy for an existing user to add PgBouncer to thePgStacDatabaseconstruct then fail to use it when creating the downstream application constructs. By making the breaking change it should force users to at least re-specify the argument name (but since the argument can be either an RDS instance or an EC2 Instance they could easily still mistakenly specify the RDS instance :/). Maybe it is not worth a breaking change - what do you think??I opted to make it a non-breaking change and will make some updates in eoapi-template to show how to use the new
pgbouncerfeature.