diff --git a/ci/vale/dictionary.txt b/ci/vale/dictionary.txt index 4e52df882ef..20ef62b350f 100644 --- a/ci/vale/dictionary.txt +++ b/ci/vale/dictionary.txt @@ -1022,6 +1022,8 @@ hybla Hydrolix hykes hypercorn +Hyperdisk +Hyperdisks hyperefficient HyperLogLog hyperparameter diff --git a/docs/assets/1238-dovecot_10-auth.conf.txt b/docs/assets/1238-dovecot_10-auth.conf.txt index fc4d23f82f9..7eb6ac624e5 100644 --- a/docs/assets/1238-dovecot_10-auth.conf.txt +++ b/docs/assets/1238-dovecot_10-auth.conf.txt @@ -118,10 +118,10 @@ auth_mechanisms = plain login #!include auth-deny.conf.ext #!include auth-master.conf.ext -!include auth-system.conf.ext +#!include auth-system.conf.ext !include auth-sql.conf.ext #!include auth-ldap.conf.ext #!include auth-passwdfile.conf.ext #!include auth-checkpassword.conf.ext #!include auth-vpopmail.conf.ext -#!include auth-static.conf.ext \ No newline at end of file +#!include auth-static.conf.ext diff --git a/docs/guides/applications/containers/build-a-cloud-native-private-registry-with-quay/index.md b/docs/guides/applications/containers/build-a-cloud-native-private-registry-with-quay/index.md index e55b891d11b..273bdda0439 100644 --- a/docs/guides/applications/containers/build-a-cloud-native-private-registry-with-quay/index.md +++ b/docs/guides/applications/containers/build-a-cloud-native-private-registry-with-quay/index.md @@ -9,7 +9,7 @@ keywords: ['build cloud-native container registry with quay','red hat quay','cen license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' --- -Docker doesn’t provide long term storage or image distribution capabilities, so developers need something more. [Docker Registry](https://docs.docker.com/registry/) performs these tasks, and using it guarantees the same application runtime environment through virtualization. However, building an image can involve a significant time investment, which is where [Quay](https://www.redhat.com/en/resources/quay-datasheet) (pronounced *kway*) comes in. A registry like Quay can both build and store containers. You can then deploy these containers in a shorter time and with less effort than using Docker Registry. This guide explains how Quay can be an essential part of the development process and details how to deploy a Quay registry. +Docker doesn’t provide long term storage or image distribution capabilities, so developers need something more. [Docker Registry](https://docs.docker.com/registry/) performs these tasks, and using it guarantees the same application runtime environment through virtualization. However, building an image can involve a significant time investment, which is where [Quay](https://www.redhat.com/en/resources/quay-datasheet) comes in. A registry like Quay can both build and store containers. You can then deploy these containers in a shorter time and with less effort than using Docker Registry. This guide explains how Quay can be an essential part of the development process and details how to deploy a Quay registry. ## What is Red Hat Quay? diff --git a/docs/guides/databases/general/database-solutions/index.md b/docs/guides/databases/general/database-solutions/index.md index 41d067f321c..78d989bb092 100644 --- a/docs/guides/databases/general/database-solutions/index.md +++ b/docs/guides/databases/general/database-solutions/index.md @@ -128,4 +128,4 @@ There are many installation and configuration guides available on our docs site - [Apache Cassandra guides](/docs/guides/databases/cassandra/) - [Redis guides](/docs/guides/databases/redis/) - [PostgreSQL guides](/docs/guides/databases/postgresql/) -- [CouchDB guides](/docs/guides/databases/couchdb/) \ No newline at end of file +- [CouchDB guides](/docs/guides/databases/couchdb/) diff --git a/docs/guides/email/postfix/email-with-postfix-dovecot-and-mysql/index.md b/docs/guides/email/postfix/email-with-postfix-dovecot-and-mysql/index.md index 9ff755ef4b3..d8b669765ca 100644 --- a/docs/guides/email/postfix/email-with-postfix-dovecot-and-mysql/index.md +++ b/docs/guides/email/postfix/email-with-postfix-dovecot-and-mysql/index.md @@ -575,7 +575,7 @@ disable_plaintext_auth = yes ... auth_mechanisms = plain login ... -!include auth-system.conf.ext +#!include auth-system.conf.ext ... !include auth-sql.conf.ext ... @@ -870,4 +870,4 @@ spamassassin unix - n n - - pipe 1. Restart the Postfix email server to get your new anti-spam settings in place: - sudo systemctl restart postfix \ No newline at end of file + sudo systemctl restart postfix diff --git a/docs/guides/kubernetes/ai-chatbot-and-rag-pipeline-for-inference-on-lke/index.md b/docs/guides/kubernetes/ai-chatbot-and-rag-pipeline-for-inference-on-lke/index.md index 539fc79d491..a0e4b791240 100644 --- a/docs/guides/kubernetes/ai-chatbot-and-rag-pipeline-for-inference-on-lke/index.md +++ b/docs/guides/kubernetes/ai-chatbot-and-rag-pipeline-for-inference-on-lke/index.md @@ -1,11 +1,11 @@ --- slug: ai-chatbot-and-rag-pipeline-for-inference-on-lke -title: "Deploy an AI Chatbot and RAG Pipeline for Inferencing on LKE" +title: "Deploy a Chatbot and RAG Pipeline for AI Inferencing on LKE" description: "Utilize the Retrieval-Augmented Generation technique to supplement an LLM with your own custom data." authors: ["Linode"] contributors: ["Linode"] published: 2025-02-11 -modified: 2025-02-13 +modified: 2025-03-11 keywords: ['kubernetes','lke','ai','inferencing','rag','chatbot','architecture'] tags: ["kubernetes","lke"] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' @@ -37,7 +37,7 @@ Follow this tutorial to deploy a RAG pipeline on Akamai’s LKE service using ou - **Kubeflow Pipeline:** Used to deploy pipelines, reusable machine learning workflows built using the Kubeflow Pipelines SDK. In this tutorial, a pipeline is used to run LlamaIndex to process the dataset and store embeddings. - **Meta’s Llama 3 LLM:** The [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) model is used as the LLM. You should review and agree to the licensing agreement before deploying. - **Milvus:** Milvus is an open-source vector database and is used for generative AI workloads. This tutorial uses Milvus to store embeddings generated by LlamaIndex and make them available to queries sent to the Llama 3 LLM. -- **Open WebUI:** This is an self-hosted AI chatbot application that’s compatible with LLMs like Llama 3 and includes a built-in inference engine for RAG solutions. Users interact with this interface to query the LLM. This can be configured to send queries straight to Llama 3 or to first load data from Milvus and send that context along with the query. +- **Open WebUI:** This is a self-hosted AI chatbot application that’s compatible with LLMs like Llama 3 and includes a built-in inference engine for RAG solutions. Users interact with this interface to query the LLM. This can be configured to send queries straight to Llama 3 or to first load data from Milvus and send that context along with the query. ## Prerequisites @@ -45,7 +45,7 @@ This tutorial requires you to have access to a few different services and local - A [Cloud Manager](https://cloud.linode.com/) account is required to use many of Akamai’s cloud computing services, including LKE. - A [Hugging Face](https://huggingface.co/) account is used for deploying the Llama 3 LLM to KServe. -- You should have both [kubectl](https://kubernetes.io/docs/reference/kubectl/) and [Helm](https://helm.sh/) installed on your local machine. These apps are used for managing your LKE cluster and installing applications to your cluster. +- You should have [kubectl](https://kubernetes.io/docs/reference/kubectl/), [Kustomize](https://kustomize.io/), and [Helm](https://helm.sh/) installed on your local machine. These apps are used for managing your LKE cluster and installing applications to your cluster. - A **custom dataset** is needed, preferably in Markdown format, though you can use other types of data if you modify the LlamaIndex configuration provided in this tutorial. This dataset should contain all of the information you want used by the Llama 3 LLM. This tutorial uses a Markdown dataset containing all of the Linode Docs. {{< note type="warning" title="Production workloads" >}} @@ -61,7 +61,7 @@ It’s not part of the scope of this document to cover the setup required to sec The first step is to provision the infrastructure needed for this tutorial and configure it with kubectl, so that you can manage it locally and install software through helm. As part of this process, we’ll also need to install the NVIDIA GPU operator at this step so that the NVIDIA cards within the GPU worker nodes can be used on Kubernetes. -1. **Provision an LKE cluster.** We recommend using at least 3 **RTX4000 Ada x1 Medium** GPU plans (plan ID: `g2-gpu-rtx4000a1-m`), though you can adjust this as needed. For reference, Kubeflow recommends 32 GB of RAM and 16 CPU cores for just their own application. This tutorial has been tested using Kubernetes v1.31, though other versions should also work. To learn more about provisioning a cluster, see the [Create a cluster](https://techdocs.akamai.com/cloud-computing/docs/create-a-cluster) guide. +1. **Provision an LKE cluster.** We recommend using at least 2 **RTX4000 Ada x1 Medium** GPU plans (plan ID: `g2-gpu-rtx4000a1-m`), though you can adjust this as needed. For reference, Kubeflow recommends 32 GB of RAM and 16 CPU cores for just their own application. This tutorial has been tested using Kubernetes v1.31, though other versions should also work. To learn more about provisioning a cluster, see the [Create a cluster](https://techdocs.akamai.com/cloud-computing/docs/create-a-cluster) guide. {{< note noTitle=true >}} GPU plans are available in a limited number of data centers. Review the [GPU product documentation](https://techdocs.akamai.com/cloud-computing/docs/gpu-compute-instances#availability) to learn more about availability. @@ -97,10 +97,10 @@ Next, let’s deploy Kubeflow on the LKE cluster. These instructions deploy all openssl rand -base64 18 ``` - 1. Create a hash of this password, replacing PASSWORD with the password generated in the previous step. This outputs a string starting with `$2y$12$`, which is password hash. + 1. Create a hash of this password, replacing PASSWORD with the password generated in the previous step. This outputs the password hash, which starts with `$2y$12$`. ```command - htpasswd -bnBC 12 "" | tr -d ':\n' + htpasswd -bnBC 12 "" PASSWORD | tr -d ':\n' ``` 1. Edit the `common/dex/base/dex-passwords.yaml` file, replacing the value for `DEX_USER_PASSWORD` with the password hash generated in the previous step. @@ -111,7 +111,7 @@ Next, let’s deploy Kubeflow on the LKE cluster. These instructions deploy all while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 20; done ``` -1. This may take some time to finish. Once it’s complete, verify that all pods are in the ready state. +1. This may take some time to finish. Once it’s complete, verify that all pods are in the running state. ```command kubectl get pods -A @@ -152,6 +152,7 @@ After Kubeflow has been installed, we can now deploy the Llama 3 LLM to KServe. name: huggingface-llama3 spec: predictor: + minReplicas: 1 model: modelFormat: name: huggingface @@ -202,6 +203,11 @@ Milvus, the vector database designed for AI inference workloads, will be used as nvidia.com/gpu: "1" limits: nvidia.com/gpu: "1" + persistentVolumeClaim: + size: 5Gi + minio: + persistence: + size: 50Gi ``` 1. Add Milvus to Helm. @@ -214,7 +220,7 @@ Milvus, the vector database designed for AI inference workloads, will be used as 1. Install Milvus using Helm. ```command - helm install my-release milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false -f milvus-custom-values.yaml + helm install my-release milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsarv3.enabled=false -f milvus-custom-values.yaml ``` ## Set up Kubeflow Pipeline to ingest data @@ -335,19 +341,19 @@ This tutorial employs a Python script to create the YAML file used within Kubefl ![Screenshot of the "New Experiment" page within Kubeflow](kubeflow-new-experiment.jpg) -1. Next, navigate to Pipelines > Pipelines and click the **Upload Pipeline** link. Select **Upload a file** and use the **Choose file** dialog box to select the pipeline YAML file that was created in a previous step. +1. Next, navigate to Pipelines > Pipelines and click the **Upload Pipeline** link. Select **Upload a file** and use the **Choose file** dialog box to select the pipeline YAML file that was created in a previous step. Click the **Create** button to create the pipeline. ![Screenshot of the "New Pipeline" page within Kubeflow](kubeflow-new-pipeline.jpg) -1. Navigate to the Pipelines > Runs page and click **Create Run**. Within the Run details section, select the pipeline and experiment that you just created. Choose *One-off* as the **Run Type** and provide the collection name and URL of the dataset (the zip file with the documents you wish to process) in the **Run parameters** section. For this tutorial, we are using `linode_docs` as the name and `https://github.com/linode/docs/archive/refs/tags/v1.360.0.zip` and the dataset URL. +1. Navigate to the Pipelines > Runs page and click **Create Run**. Within the Run details section, select the pipeline and experiment that you just created. Choose *One-off* as the **Run Type** and provide the collection name and URL of the dataset (the zip file with the documents you wish to process) in the **Run parameters** section. For this tutorial, we are using `linode_docs` as the name and `https://github.com/linode/docs/archive/refs/tags/v1.366.0.zip` as the dataset URL. ![Screenshot of the "Start a new run" page within Kubeflow](kubeflow-new-run.jpg) -1. Click **Start** to run the pipeline. This process takes some time. For reference, it took ~10 minutes for the run to complete successfully on the linode.com/docs dataset. +1. Click **Start** to run the pipeline. This process takes some time. For reference, it takes about ~10 minutes for the run to complete on the linode.com/docs dataset. ## Deploy the chatbot -To finish up this tutorial, we will install the Open-WebUI chatbot and configure it to connect the data generated in the Kubernetes Pipeline with the LLM deployed in KServe. Once this is up and running, you can open up a browser interface to the chatbot and ask it questions. Chatbot UI will use the Milvus database to load context related to the search and send it, along with your query, to the Llama 3 instance within KServe. The LLM will send back a response to the chatbot and your browser will display an answer that is informed by your own custom data. +To finish up this tutorial, install the Open-WebUI chatbot and configure it to connect the data generated in the Kubernetes Pipeline with the LLM deployed in KServe. Once this is up and running, you can open up a browser interface to the chatbot and ask it questions. Chatbot UI uses the Milvus database to load context related to the search and sends it, along with your query, to the Llama 3 instance within KServe. The LLM then sends back a response to the chatbot and your browser displays an answer that is informed by your own custom data. ### Create the RAG pipeline files @@ -360,6 +366,7 @@ Despite the naming, these RAG pipeline files are not related to the Kubeflow pip ```file {title="pipeline-requirements.txt"} requests pymilvus + opencv-python-headless llama-index llama-index-vector-stores-milvus llama-index-embeddings-huggingface @@ -368,7 +375,7 @@ Despite the naming, these RAG pipeline files are not related to the Kubeflow pip 1. Create a file called `rag_pipeline.py` with the following contents. The filenames of both the `pipeline-requirements.txt` and `rag_pipeline.py` files should not be changed as they are referenced within the Open WebUI Pipeline configuration file. - ```file {title="rag-pipeline.py"} + ```file {title="rag_pipeline.py"} """ title: RAG Pipeline version: 1.0 @@ -594,4 +601,6 @@ Now that the chatbot has been configured, the final step is to access the chatbo - The **RAG Pipeline** model that you defined in a previous section does use data from your custom dataset. Ask it a question relevant to your data and the chatbot should respond with an answer that is informed by the custom dataset you configured. - ![Screenshot of a RAG Pipeline query in Open WebUI](open-webui-rag-pipeline.jpg) \ No newline at end of file + ![Screenshot of a RAG Pipeline query in Open WebUI](open-webui-rag-pipeline.jpg) + + The response time depends on a variety of factors. Using similar cluster resources and the same dataset as this guide, an estimated response time is between 6 to 70 seconds. \ No newline at end of file diff --git a/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md b/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md index 09b203067b3..ae4ff09d7a2 100644 --- a/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md +++ b/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md @@ -31,7 +31,7 @@ This guide shows how to set up a Harbor registry on a dedicated compute instance The Harbor installation in this guide assumes that you have [a domain name registered through a domain registrar](/docs/products/networking/dns-manager/get-started/#register-the-domain), and that you can edit the DNS records for this domain. This is so that SSL connections can be configured for the Harbor server. If you do not have a domain name, register one now. -The infrastructure for this guide is created on the Akamai Connected Cloud platform. If you do not already have one, [create an account](/docs/products/platform/get-started/) for the platform. +The infrastructure for this guide is created on the Akamai Cloud platform. If you do not already have one, [create an account](/docs/products/platform/get-started/) for the platform. The following is a summary of the infrastructure created in this guide. Instructions for creating these services are included later in the guide: diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-aws-ebs-to-linode-block-storage/aws-block-storage-migration-workflow.svg b/docs/guides/platform/migrate-to-linode/migrate-from-aws-ebs-to-linode-block-storage/aws-block-storage-migration-workflow.svg new file mode 100644 index 00000000000..7caa8a5a872 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/migrate-from-aws-ebs-to-linode-block-storage/aws-block-storage-migration-workflow.svg @@ -0,0 +1,164 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-aws-ebs-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-aws-ebs-to-linode-block-storage/index.md new file mode 100644 index 00000000000..7349cca991e --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/migrate-from-aws-ebs-to-linode-block-storage/index.md @@ -0,0 +1,285 @@ +--- +slug: migrate-from-aws-ebs-to-linode-block-storage +title: "Migrate From AWS EBS to Linode Block Storage" +description: "Learn how to migrate data from an AWS EBS volume to a Linode Block Storage Volume." +authors: ["Akamai"] +contributors: ["Akamai"] +published: 2025-03-07 +keywords: ['migrate','aws ebs','linode block storage'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +--- + +This guide describes the process of migrating a single volume from AWS Elastic Block Store (EBS) to Linode Block Storage using the rsync file synchronization utility. This guide focuses on migrating secondary EBS persistent data storage volumes rather than migrating the root volume of an AWS EC2 instance. + +## Block Storage Migration Workflow Diagram + +![AWS EBS to Linode Block Storage Migration Workflow Diagram](aws-block-storage-migration-workflow.svg?diagram-description-id=aws-linode-bs-migration) + +1. The rsync command is run from a Linode instance and connects to an AWS EC2 instance. + +2. The EC2 instance sends data on the EBS volume to a Block Storage Volume attached to the Linode instance via an established rsync connection. + + 1. Egress costs for the migrated data are measured when the data leaves the AWS platform. These costs are billed by AWS. +{#aws-linode-bs-migration .large-diagram} + +## Linode Block Storage vs. AWS EBS + +When an AWS EC2 instance is first created, AWS creates a root volume from an Amazon Machine Image (AMI) on the EBS service. This EBS volume is attached to the instance and contains the operating system. One or more secondary EBS volumes can also be created with the EBS service for temporary files and for persistent data. + +Like EBS, Linode Block Storage also provides block-level storage volumes to be used with virtual machines. Unlike EBS, Linode Block Storage is generally used for persistent data rather than operating system, boot disks, or temporary data. These other roles are fulfilled by a Linode instance's bundled disk, which is stored on the same host as the Compute Instance. Linode's bundled disk storage is also more suitable for applications that feature high disk operations, like high-traffic databases. + +{{< note >}} +EC2 root volumes can also be created on AWS's [instance store](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html), but these volumes are destroyed when the instance is shut down (or fails). Bundled disk storage for Linode instances persists between shut downs and reboots, and is only removed when the Linode instance is destroyed. +{{< /note >}} + +## Migration Considerations + +The following are important time, cost, and security considerations to keep in mind when migrating your EBS volumes to Linode Block Storage. + +### Migration Time Estimates + +The time it takes to migrate a volume is a function of the data stored on that volume, which can be substantial for larger migrations. To determine how much data is stored on your volume, run the `df` command from your EC2 instance: + +```command {title="SSH session with AWS EC2 instance"} +df -h +``` + +Your data volume should appear, and the `Used` column shows how much data is stored on the volume. In this example, the filesystem location for the volume is `/dev/sdb/`, but your location may be different: + +```output +Filesystem Size Used Avail Use% Mounted on +/dev/sdb 20G 4.4G 16G 23% /data +``` + +Bandwidth for the transfer can vary according to different factors, including: + +- Outbound bandwidth limits for your EC2 instance +- Geographic distance between the EC2 instance and the Linode instance +- Disk operation limits + +When planning your migration, consider performing a bandwidth test between the two locations first. Then, use the observed bandwidth from the test to calculate the estimated migration time for the volume. + +Utilities like [iperf](/docs/guides/install-iperf-to-diagnose-network-speed-in-linux/) can be useful for performing this type of bandwidth measurement. Alternatively, you can create a test file on the EC2 instance, migrate it following the [instructions](#block-storage-migration-instructions) in this guide, and then view the bandwidth reported by rsync's output. + +You can use the `dd` command to generate a sample 1GB test file: + +```command {title="SSH session with AWS EC2 instance"} +sudo dd if=/dev/zero of=/data/dummyfile bs=1M count=1024 +``` + +### Migration Egress Costs + +The cost to migrate a volume is a function of the data stored on that volume, which can be substantial for larger migrations. These costs are incurred as egress fees when the data leaves the AWS platform and are billed by AWS. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the volume, and review the Data Transfer pricing in [AWS's EC2 pricing documentation](https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer) for assistance with calculating this amount. + +{{< note >}} +Egress costs for migrations out of the AWS platform *may* be waived by AWS. Review this [AWS blog post](https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/) for more information on this scenario. +{{< /note >}} + +Inbound traffic sent to your Linode instance and Block Storage Volume have no fees incurred on the Akamai Cloud platform. + +### Security and Firewalls + +For data security reasons, files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted by default. + +Both your EC2 and Akamai Cloud firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and EC2 instance. + +## Block Storage Migration Instructions + +### Prerequisites and Assumptions + +This guide assumes that you have an EC2 instance running Linux. The guide also assumes you have a secondary EBS volume attached to the instance in addition to the instance's root volume. The assumed filesystem path for this EBS volume is `/data`, and the username for the EC2 instance is `awsuser`. + +### Prepare a Linode Block Storage Volume + +1. To transfer data to a Linode Block Storage volume, it must first be attached to a Linode instance. You may create a new Linode instance for the purpose of this migration ([Create a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance)). Alternatively, you can use an existing Linode instance for the migration. + + {{< note >}} + If you create an instance to use for this migration, you may wish to delete it after the migration is complete. Deleting an instance that has an attached volume does not delete the volume. + {{< /note >}} + +1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume should have a capacity equal to or higher than the total data stored on the source EBS volume. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the volume. + + When creating the volume, Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/linode-block-storage-volume`). + +### Configure Firewalls + +In this guide, the rsync command is run from a Linode instance and connects to an EC2 instance. This means that the EC2 instance should accept inbound SSH traffic (port 22). You may also wish to specifically add the IP address of the Linode instance to the allow list for inbound traffic of the EC2 instance. + +Linux distributions (on both Linode instances and EC2 instances) can have software firewalls configured inside the instance. The following guides describe some software firewalls that your instances may use: + +- [Configure a Firewall with Firewalld](/docs/guides/introduction-to-firewalld-on-centos/) +- [How to Configure a Firewall with UFW](/docs/guides/configure-firewall-with-ufw/) +- [A Tutorial for Controlling Network Traffic with iptables](/docs/guides/control-network-traffic-with-iptables/) + +You may also configure Cloud Firewalls to control traffic before it arrives at your Linode instance. Our [Cloud Firewall](https://techdocs.akamai.com/cloud-computing/docs/cloud-firewall/) product documentation describes how to configure these rules. The [Comparing Cloud Firewalls to Linux firewall software](https://techdocs.akamai.com/cloud-computing/docs/comparing-cloud-firewalls-to-linux-firewall-software) guide further describes the difference between network firewalls and software firewalls. [AWS's product documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-security-groups.html) describes how to configure *security groups* for EC2 instances which function as a network firewall. + +### Configure SSH Key Pair + +This guide uses SSH public key authentication for the rsync connection. You must have a public and private key pair installed on the Linode instance and EC2 instance. The [Generate an SSH Key Pair](/docs/guides/use-public-key-authentication-with-ssh/#generate-an-ssh-key-pair) section of the [SSH Public Key Authentication](/docs/guides/use-public-key-authentication-with-ssh/) guide describes how to create and use a key pair. + +This guide assumes the public and private keys are named `id_rsa.pub` and `id_rsa`, but your keys may have different names depending on the type of key pair you are using. + +- The *public key* should be uploaded to the EC2 instance. It should be appended to a new line of the `authorized_keys` file of the user on the EC2 instance (e.g. `/home/awsuser/.ssh/authorized_keys`). + +- The *private key* should be located on the Linode instance. It should be uploaded to the `.ssh/` directory of the user on the Linode instance (e.g. `/home/linodeuser/.ssh/`) and have permissions set to `600`: + + ```command {title="SSH session with Linode instance"} + chmod 600 /home/linodeuser/.ssh/id_rsa + ``` + +### Initiate the Migration + +These instructions implement two recommended practices: + +- Running rsync in a persistent process + +- Sending output and errors to log files + +Migrations can take a long time, so having them run independently of your SSH session is important. This guide uses `tmux` to create a terminal session that persists between SSH connections. By sending output and errors to log files, you can keep a record of any migration failures that may happen. + +Review our [tmux guide](/docs/guides/persistent-terminal-sessions-with-tmux/) for help with other tmux commands. + +1. Install the `tmux` utility on your Linode instance using the official tmux instructions: [Installing tmux](https://github.com/tmux/tmux/wiki/Installing#installing-tmux). + +1. Create a new tmux session named `block-storage-migration`. This session is used to initiate the migration: + + ```command {title="SSH session with Linode instance"} + tmux new -s block-storage-migration + ``` + + After running this command, the tmux session is immediately activated in your terminal. + +1. Run the following commands to start migrating the contents of your EBS volume to your Linode Block Storage Volume: + + ```command {title="SSH session with Linode instance (bs-migration tmux session)"} + echo "\n\nInitiating migration $(date)\n---" | tee -a bs-migration-logs.txt bs-migration-errors.txt >/dev/null + + rsync -chavzP --stats -e "ssh -i /home/linodeuser/.ssh/id_rsa" {{< placeholder "awsuser" >}}@{{< placeholder "EC2_INSTANCE_IP" >}}:/data/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt + ``` + + Replace the following values with the actual values from your EC2 instance and Linode instance: + + - `/home/linodeuser/.ssh/id_rsa`: The name and location of the private key on your Linode instance + - `{{< placeholder "awsuser" >}}`: The name of the user on the EC2 instance + - `{{< placeholder "EC2_INSTANCE_IP" >}}`: The IP address of the EC2 instance + - `/data/`: The directory under which the EBS volume is mounted + - `/mnt/linode-block-storage-volume`: The directory under which your Linode volume is mounted + + {{< note >}} + You may be prompted to accept the host key of the EC2 instance if it is the first time that the Linode has connected to it. + {{< /note >}} + + **Command breakdown**: + + The first `echo` appends a message to the log files. Below is a detailed explanation of the key flags and parameters used in the `rsync` command: + + - `-c`: Tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but `-c` forces it to compute checksums. This is slower but can be more accurate if you want to be sure that files match exactly. + + - `-h`: Human-readable output, which makes file sizes like transfer statistics easier to read by using units like KB and MB, rather than raw byte counts. + + - `-a`: Archive mode. This is equivalent to specifying: `-rlptgoD`. The result of the `-a` flag is a complete, near-exact copy of the source directory: + + - `-r`: Recursively copy directories + + - `-l`: Preserve symbolic links + + - `-p`: Retain file permissions + + - `-t`: Keep timestamps + + - `-g`: Preserve group ownership + + - `-o`: Maintain file ownership + + - `-D`: Retain device files and special files + + - `-v`: Verbose mode. This makes rsync output more detailed information about what it is doing, and can be helpful for monitoring the progress of a large transfer or troubleshooting. + + - `-z`: Compression. This enables compression during data transfer, which can save bandwidth and speed up the transfer if the network connection is relatively slow. + + - `-P`: Combines two other flags: + + - `--progress`, which displays progress information for each file transfer. + + - `--partial`, which keeps partially transferred files if the transfer is interrupted, allowing it to resume more easily next time. + + - `--stats`: Provides detailed statistics at the end of the transfer, such as total bytes transferred, transfer speed, and file counts. + + - `-e "ssh -i /home/linodeuser/.ssh/id_rsa"`: Specifies a remote shell (SSH) with an identity key file for authentication. + + - `{{< placeholder "awsuser" >}}@{{< placeholder "EC2_INSTANCE_IP" >}}:/data/`: This specifies the source directory you're syncing from: + + - `{{< placeholder "awsuser" >}}`: The username on the remote server. + + - `{{< placeholder "EC2_INSTANCE_IP" >}}`: The IP address of the remote server. + + - `/data/`: The path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /data, rather than creating a /data directory in the target. + + - `/mnt/linode-block-storage-volume`: The destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /data contents here. + +### Monitor the Migration + +Because the stdout and stderr streams were redirected to log files, the rsync command will not produce output in the terminal. Follow these steps to inspect and monitor the contents of the logs: + +1. To avoid interrupting the rsync process, *detach* from the tmux session by entering the following sequence of keystrokes: Ctrl + B followed by D. You are returned to the SSH session that created the tmux session: + + ```output + [detached (from session block-storage-migration)] + ``` + +1. Use `tail -f` to inspect the log and error files and monitor any new output from them: + + ```command {title="SSH session with Linode instance"} + tail -f block-storage-migration-logs.txt + ``` + + ```command {title="SSH session with Linode instance"} + tail -f block-storage-migration-errors.txt + ``` + + Enter Ctrl + C to stop `tail`. + +1. You can re-enter the tmux session with the `tmux attach` command: + + ```command {title="SSH session with Linode instance"} + tmux attach -t block-storage-migration + ``` + +### Verify the Migration + +To verify that rsync has synced all the files as expected, re-run the `rsync` command with the `--dry-run –-stats` flags, replacing the same values as before: + +```command {title="SSH session with Linode instance"} +rsync -chavzP --stats --dry-run -e "ssh -i /home/awsuser/.ssh/id_rsa" {{< placeholder "awsuser" >}}@{{< placeholder "EC2_INSTANCE_IP" >}}:/data/ /mnt/linode-block-storage-volume +``` + +If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A previous successful rsync transfer should result in the following output. Note that the number of created, deleted, and transferred files are zero: + +```output +receiving incremental file list + +Number of files: 2 (reg: 1, dir: 1) +Number of created files: 0 +Number of deleted files: 0 +Number of regular files transferred: 0 +Total file size: 10.49M bytes +Total transferred file size: 0 bytes +Literal data: 0 bytes +Matched data: 0 bytes +File list size: 84 +File list generation time: 0.003 seconds +File list transfer time: 0.000 seconds +Total bytes sent: 20 +Total bytes received: 95 + +sent 20 bytes received 95 bytes 230.00 bytes/sec +total size is 10.49M speedup is 91,180.52 (DRY RUN) +``` + +### Cleanup after Migration + +After the migration is complete, you may determine that the EC2 instance and Linode instance no longer need to communicate. You can close traffic between these servers by doing the following: + +- Remove the firewall access granted in the [Configure Firewalls](#configure-firewalls) section + +- Revoke the SSH key used for the transfer. This is done by removing the SSH public key that was referenced from the `/home/awsuser/.ssh/authorized_keys` file on the EC2 instance. \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg index a773bf91145..0a914e78095 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg @@ -1,179 +1,178 @@ - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" xml:space="preserve" + xmlns:serif="http://www.serif.com/" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:2;"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index d578ed9cbe9..25f0741a2f2 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -1,12 +1,11 @@ --- slug: migrate-from-azure-disk-storage-to-linode-block-storage title: "Migrate From Azure Disk Storage to Linode Block Storage" -description: "Two to three sentences describing your guide." -og_description: "Optional two to three sentences describing your guide when shared on social media. If omitted, the `description` parameter is used within social links." +description: "Learn how to migrate data from an Azure Disk Storage managed disk to a Linode Block Storage Volume." authors: ["Leon Yen","Nathan Melehan"] contributors: ["Leon Yen","Nathan Melehan"] published: 2024-11-18 -keywords: ['list','of','keywords','and key phrases'] +keywords: ['migrate','azure disk storage','linode block storage'] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' --- @@ -20,8 +19,7 @@ This guide describes the process of migrating a single volume from Azure Disk St 2. The Azure VM sends data on the Azure data disk to a Block Storage Volume attached to the Linode instance via an established rsync connection. - 2a. Egress costs for the migrated data are measured when the data leaves the Azure platform. These costs are billed by Azure. - + 1. Egress costs for the migrated data are measured when the data leaves the Azure platform. These costs are billed by Azure. {#azure-linode-bs-migration .large-diagram} ## Linode Block Storage vs. Azure Disk Storage diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage/gcp-block-storage-migration-workflow.svg b/docs/guides/platform/migrate-to-linode/migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage/gcp-block-storage-migration-workflow.svg new file mode 100644 index 00000000000..ae122322bc5 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage/gcp-block-storage-migration-workflow.svg @@ -0,0 +1,191 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage/index.md new file mode 100644 index 00000000000..0a686185a39 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage/index.md @@ -0,0 +1,283 @@ +--- +slug: migrate-from-gcp-hyperdisk-and-persistent-disk-to-linode-block-storage +title: "Migrate from GCP Hyperdisk and Persistent Disk to Linode Block Storage" +description: "Learn how to migrate data from a Hyperdisk or Persistent Disk on Google Cloud Platform to a Linode Block Storage Volume." +authors: ["Akamai"] +contributors: ["Akamai"] +published: 2025-03-07 +keywords: ['migrate','gcp hyperdisk','gcp persistent disk','linode block storage'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +--- + +This guide describes the process of migrating a single disk from Google Cloud Platform's (GCP) Hyperdisk or Persistent Disk services to Linode Block Storage using the rsync file synchronization utility. This guide focuses on migrating secondary persistent data storage disks rather than migrating the boot disk of a Google Compute Engine instance. + +## Block Storage Migration Workflow Diagram + +![GCP Hyperdisk and Persistent Disk to Linode Block Storage Migration Workflow Diagram](gcp-block-storage-migration-workflow.svg?diagram-description-id=gcp-linode-bs-migration) + +1. The rsync command is run from a Linode instance and connects to a Google Compute Engine instance. + +2. The Compute Engine instance sends data on the Hyperdisk or Persistent Disk to a Block Storage Volume attached to the Linode instance via an established rsync connection. + + 1. Egress costs for the migrated data are measured when the data leaves the GCP platform. These costs are billed by Google. +{#gcp-linode-bs-migration .large-diagram} + +## Linode Block Storage vs. GCP Hyperdisk and Persistent Disk + +Hyperdisk and Persistent Disk are Google's persistent network-attached block storage services. They have different performance characteristics and are available to different tiers of Compute Engine instances, but otherwise serve the similar functions as a block storage device. + +When a Google Compute Engine instance is first created, Google creates a boot disk on the Hyperdisk or Persistent Disk services. This disk is attached to the instance and contains the operating system. One or more secondary disks can also be created with theses services for temporary files and for persistent data. + +Like Hyperdisk and Persistent Disk, Linode Block Storage also provides block-level storage volumes to be used with virtual machines. Unlike Hyperdisk and Persistent Disk, Linode Block Storage is generally used for persistent data rather than operating system, boot disks, or temporary data. These other roles are fulfilled by a Linode instance's bundled disk, which is stored on the same host as the Compute Instance. Linode's bundled disk storage is also more suitable for applications that feature high disk operations, like high-traffic databases. + +## Migration Considerations + +The following are important time, cost, and security considerations to keep in mind when migrating your Hyperdisks and Persistent Disks to Linode Block Storage. + +### Migration Time Estimates + +The time it takes to migrate a disk is a function of the data stored on that disk, which can be substantial for larger migrations. To determine how much data is stored on your disk, run the `df` command from your Compute Engine instance: + +```command {title="SSH session with Google Compute Engine instance"} +df -h +``` + +Your data disk should appear, and the `Used` column shows how much data is stored on the disk. In this example, the filesystem location for the disk is `/dev/sdb/`, but your location may be different: + +```output +Filesystem Size Used Avail Use% Mounted on +/dev/sdb 20G 4.4G 16G 23% /mnt/disks/data +``` + +Bandwidth for the transfer can vary according to different factors, including: + +- Outbound bandwidth limits for your Compute Engine instance +- Geographic distance between the Compute Engine instance and the Linode instance +- Disk operation limits + +When planning your migration, consider performing a bandwidth test between the two locations first. Then, use the observed bandwidth from the test to calculate the estimated migration time for the disk. + +Utilities like [iperf](/docs/guides/install-iperf-to-diagnose-network-speed-in-linux/) can be useful for performing this type of bandwidth measurement. Alternatively, you can create a test file on the Compute Engine instance, migrate it following the [instructions](#block-storage-migration-instructions) in this guide, and then view the bandwidth reported by rsync's output. + +You can use the `dd` command to generate a sample 1GB test file: + +```command {title="SSH session with Google Compute Engine instance"} +sudo dd if=/dev/zero of=/mnt/disks/data/dummyfile bs=1M count=1024 +``` + +### Migration Egress Costs + +The cost to migrate a disk is a function of the data stored on that disk, which can be substantial for larger migrations. These costs are incurred as egress fees when the data leaves GCP and are billed by Google. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk, and review the Data Transfer pricing in [Google's Compute Engine pricing documentation](https://cloud.google.com/network-tiers/pricing) for assistance with calculating this amount. + +{{< note >}} +Egress costs for migrations out of GCP *may* be waived by Google. Review this [GCP documentation](https://cloud.google.com/exit-cloud?hl=en) for more information on this scenario. +{{< /note >}} + +Inbound traffic sent to your Linode instance and Block Storage Volume have no fees incurred on the Akamai Cloud platform. + +### Security and Firewalls + +For data security reasons, files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted by default. + +Both your Compute Engine and Akamai Cloud firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and Compute Engine instance. + +## Block Storage Migration Instructions + +### Prerequisites and Assumptions + +This guide assumes that you have a Compute Engine instance running Linux. The guide also assumes you have a secondary Hyperdisk or Persistent Disk volume attached to the instance in addition to the instance's boot disk. The assumed filesystem path for the volume is `/mnt/disks/data`, and the username for the Compute Engine instance is `gcpuser`. + +### Prepare a Linode Block Storage Volume + +1. To transfer data to a Linode Block Storage volume, it must first be attached to a Linode instance. You may create a new Linode instance for the purpose of this migration ([Create a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance)). Alternatively, you can use an existing Linode instance for the migration. + + {{< note >}} + If you create an instance to use for this migration, you may wish to delete it after the migration is complete. Deleting an instance that has an attached volume does not delete the volume. + {{< /note >}} + +1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume should have a capacity equal to or higher than the total data stored on the source Hyperdisk or Persistent Disk volume. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the volume. + + When creating the volume, Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/linode-block-storage-volume`). + +### Configure Firewalls + +In this guide, the rsync command is run from a Linode instance and connects to a Compute Engine instance. This means that the Compute Engine instance should accept inbound SSH traffic (port 22). You may also wish to specifically add the IP address of the Linode instance to the allow list for inbound traffic of the Compute Engine instance. + +Linux distributions (on both Linode instances and Compute Engine instances) can have software firewalls configured inside the instance. The following guides describe some software firewalls that your instances may use: + +- [Configure a Firewall with Firewalld](/docs/guides/introduction-to-firewalld-on-centos/) +- [How to Configure a Firewall with UFW](/docs/guides/configure-firewall-with-ufw/) +- [A Tutorial for Controlling Network Traffic with iptables](/docs/guides/control-network-traffic-with-iptables/) + +You may also configure Cloud Firewalls to control traffic before it arrives at your Linode instance. Our [Cloud Firewall](https://techdocs.akamai.com/cloud-computing/docs/cloud-firewall/) product documentation describes how to configure these rules. The [Comparing Cloud Firewalls to Linux firewall software](https://techdocs.akamai.com/cloud-computing/docs/comparing-cloud-firewalls-to-linux-firewall-software) guide further describes the difference between network firewalls and software firewalls. [GCP's product documentation](https://cloud.google.com/security/products/firewall?hl=en) describes how to configure cloud firewalls for Compute Engine instances. + +### Configure SSH Key Pair + +This guide uses SSH public key authentication for the rsync connection. You must have a public and private key pair installed on the Linode instance and Compute Engine instance. The [Generate an SSH Key Pair](/docs/guides/use-public-key-authentication-with-ssh/#generate-an-ssh-key-pair) section of the [SSH Public Key Authentication](/docs/guides/use-public-key-authentication-with-ssh/) guide describes how to create and use a key pair. + +This guide assumes the public and private keys are named `id_rsa.pub` and `id_rsa`, but your keys may have different names depending on the type of key pair you are using. + +- The *public key* should be uploaded to the Compute Engine instance. It should be appended to a new line of the `authorized_keys` file of the user on the Compute Engine instance (e.g. `/home/gcpuser/.ssh/authorized_keys`). + +- The *private key* should be located on the Linode instance. It should be uploaded to the `.ssh/` directory of the user on the Linode instance (e.g. `/home/linodeuser/.ssh/`) and have permissions set to `600`: + + ```command {title="SSH session with Linode instance"} + chmod 600 /home/linodeuser/.ssh/id_rsa + ``` + +### Initiate the Migration + +These instructions implement two recommended practices: + +- Running rsync in a persistent process + +- Sending output and errors to log files + +Migrations can take a long time, so having them run independently of your SSH session is important. This guide uses `tmux` to create a terminal session that persists between SSH connections. By sending output and errors to log files, you can keep a record of any migration failures that may happen. + +Review our [tmux guide](/docs/guides/persistent-terminal-sessions-with-tmux/) for help with other tmux commands. + +1. Install the `tmux` utility on your Linode instance using the official tmux instructions: [Installing tmux](https://github.com/tmux/tmux/wiki/Installing#installing-tmux). + +1. Create a new tmux session named `block-storage-migration`. This session is used to initiate the migration: + + ```command {title="SSH session with Linode instance"} + tmux new -s block-storage-migration + ``` + + After running this command, the tmux session is immediately activated in your terminal. + +1. Run the following commands to start migrating the contents of your Hyperdisk or Persistent Disk volume to your Linode Block Storage Volume: + + ```command {title="SSH session with Linode instance (bs-migration tmux session)"} + echo "\n\nInitiating migration $(date)\n---" | tee -a bs-migration-logs.txt bs-migration-errors.txt >/dev/null + + rsync -chavzP --stats -e "ssh -i /home/linodeuser/.ssh/id_rsa" {{< placeholder "gcpuser" >}}@{{< placeholder "COMPUTE_ENGINE_INSTANCE_IP" >}}:/mnt/disks/data/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt + ``` + + Replace the following values with the actual values from your Compute Engine instance and Linode instance: + + - `/home/linodeuser/.ssh/id_rsa`: The name and location of the private key on your Linode instance + - `{{< placeholder "gcpuser" >}}`: The name of the user on the Compute Engine instance + - `{{< placeholder "COMPUTE_ENGINE_INSTANCE_IP" >}}`: The IP address of the Compute Engine instance + - `/mnt/disks/data/`: The directory under which the Hyperdisk or Persistent Disk volume is mounted + - `/mnt/linode-block-storage-volume`: The directory under which your Linode volume is mounted + + {{< note >}} + You may be prompted to accept the host key of the Compute Engine instance if it is the first time that the Linode has connected to it. + {{< /note >}} + + **Command breakdown**: + + The first `echo` appends a message to the log files. Below is a detailed explanation of the key flags and parameters used in the `rsync` command: + + - `-c`: Tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but `-c` forces it to compute checksums. This is slower but can be more accurate if you want to be sure that files match exactly. + + - `-h`: Human-readable output, which makes file sizes like transfer statistics easier to read by using units like KB and MB, rather than raw byte counts. + + - `-a`: Archive mode. This is equivalent to specifying: `-rlptgoD`. The result of the `-a` flag is a complete, near-exact copy of the source directory: + + - `-r`: Recursively copy directories + + - `-l`: Preserve symbolic links + + - `-p`: Retain file permissions + + - `-t`: Keep timestamps + + - `-g`: Preserve group ownership + + - `-o`: Maintain file ownership + + - `-D`: Retain device files and special files + + - `-v`: Verbose mode. This makes rsync output more detailed information about what it is doing, and can be helpful for monitoring the progress of a large transfer or troubleshooting. + + - `-z`: Compression. This enables compression during data transfer, which can save bandwidth and speed up the transfer if the network connection is relatively slow. + + - `-P`: Combines two other flags: + + - `--progress`, which displays progress information for each file transfer. + + - `--partial`, which keeps partially transferred files if the transfer is interrupted, allowing it to resume more easily next time. + + - `--stats`: Provides detailed statistics at the end of the transfer, such as total bytes transferred, transfer speed, and file counts. + + - `-e "ssh -i /home/linodeuser/.ssh/id_rsa"`: Specifies a remote shell (SSH) with an identity key file for authentication. + + - `{{< placeholder "gcpuser" >}}@{{< placeholder "COMPUTE_ENGINE_INSTANCE_IP" >}}:/mnt/disks/data/`: This specifies the source directory you're syncing from: + + - `{{< placeholder "gcpuser" >}}`: The username on the remote server. + + - `{{< placeholder "COMPUTE_ENGINE_INSTANCE_IP" >}}`: The IP address of the remote server. + + - `/mnt/disks/data/`: The path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /mnt/disks/data, rather than creating a /mnt/disks/data directory in the target. + + - `/mnt/linode-block-storage-volume`: The destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /mnt/disks/data contents here. + +### Monitor the Migration + +Because the stdout and stderr streams were redirected to log files, the rsync command will not produce output in the terminal. Follow these steps to inspect and monitor the contents of the logs: + +1. To avoid interrupting the rsync process, *detach* from the tmux session by entering the following sequence of keystrokes: Ctrl + B followed by D. You are returned to the SSH session that created the tmux session: + + ```output + [detached (from session block-storage-migration)] + ``` + +1. Use `tail -f` to inspect the log and error files and monitor any new output from them: + + ```command {title="SSH session with Linode instance"} + tail -f block-storage-migration-logs.txt + ``` + + ```command {title="SSH session with Linode instance"} + tail -f block-storage-migration-errors.txt + ``` + + Enter Ctrl + C to stop `tail`. + +1. You can re-enter the tmux session with the `tmux attach` command: + + ```command {title="SSH session with Linode instance"} + tmux attach -t block-storage-migration + ``` + +### Verify the Migration + +To verify that rsync has synced all the files as expected, re-run the `rsync` command with the `--dry-run –-stats` flags, replacing the same values as before: + +```command {title="SSH session with Linode instance"} +rsync -chavzP --stats --dry-run -e "ssh -i /home/gcpuser/.ssh/id_rsa" {{< placeholder "gcpuser" >}}@{{< placeholder "COMPUTE_ENGINE_INSTANCE_IP" >}}:/mnt/disks/data/ /mnt/linode-block-storage-volume +``` + +If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A previous successful rsync transfer should result in the following output. Note that the number of created, deleted, and transferred files are zero: + +```output +receiving incremental file list + +Number of files: 2 (reg: 1, dir: 1) +Number of created files: 0 +Number of deleted files: 0 +Number of regular files transferred: 0 +Total file size: 10.49M bytes +Total transferred file size: 0 bytes +Literal data: 0 bytes +Matched data: 0 bytes +File list size: 84 +File list generation time: 0.003 seconds +File list transfer time: 0.000 seconds +Total bytes sent: 20 +Total bytes received: 95 + +sent 20 bytes received 95 bytes 230.00 bytes/sec +total size is 10.49M speedup is 91,180.52 (DRY RUN) +``` + +### Cleanup after Migration + +After the migration is complete, you may determine that the Compute Engine instance and Linode instance no longer need to communicate. You can close traffic between these servers by doing the following: + +- Remove the firewall access granted in the [Configure Firewalls](#configure-firewalls) section + +- Revoke the SSH key used for the transfer. This is done by removing the SSH public key that was referenced from the `/home/gcpuser/.ssh/authorized_keys` file on the Compute Engine instance. \ No newline at end of file diff --git a/docs/guides/security/basics/using-fail2ban-to-secure-your-server-a-tutorial/index.md b/docs/guides/security/basics/using-fail2ban-to-secure-your-server-a-tutorial/index.md index df113c85e01..a4887da8bf1 100644 --- a/docs/guides/security/basics/using-fail2ban-to-secure-your-server-a-tutorial/index.md +++ b/docs/guides/security/basics/using-fail2ban-to-secure-your-server-a-tutorial/index.md @@ -483,13 +483,7 @@ The best way to understand how failregex works is to write one. Although we do n - - \[(\d{2})/\w{3}/\d{4}: ``` -1. The next sequence is a series of two-digit numbers that make up the time. Because we defined the day of the month as a two-digit number in a capture group (the parentheses), we can backreference it using `\1` (since it is the *first* capture group). Again, the colons are literals: - - ```command - - - \[(\d{2})/\w{3}/\d{4}:\1:\1:\1 - ``` - - If you do not want to use backreferences this can also be written as: +1. The next sequence is a series of two-digit numbers that make up the time. Again, the colons are literals: ```command - - \[\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2} @@ -617,4 +611,4 @@ CentOS 7 and Fedora additionally require two extra commands to be fully stopped ```command systemctl disable --now fail2ban -``` \ No newline at end of file +``` diff --git a/docs/guides/tools-reference/tools/modify-file-permissions-with-chmod/index.md b/docs/guides/tools-reference/tools/modify-file-permissions-with-chmod/index.md index d5dd093977f..1b30d0e931d 100644 --- a/docs/guides/tools-reference/tools/modify-file-permissions-with-chmod/index.md +++ b/docs/guides/tools-reference/tools/modify-file-permissions-with-chmod/index.md @@ -307,4 +307,4 @@ sudo chmod -R 644 /var/www/html/ * [How to set permissions for the webserver directory](https://www.linode.com/community/questions/8808/setting-ownership-and-permissions-for-webserver-directories) * [Permission issues with Ubuntu](https://www.linode.com/community/questions/6076/permission-issues-with-ubuntu-amp-wordpress-chown-fails) * [Issues with /var/www permissions](https://www.linode.com/community/questions/7302/varwww-permissions) -{{}} \ No newline at end of file +{{}} diff --git a/docs/guides/uptime/analytics/plausible/index.md b/docs/guides/uptime/analytics/plausible/index.md index ec602bfde01..99b89782d79 100644 --- a/docs/guides/uptime/analytics/plausible/index.md +++ b/docs/guides/uptime/analytics/plausible/index.md @@ -9,7 +9,7 @@ keywords: ['plausible','analytics'] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' --- -[Plausible](https://plausible.io/) is a free and open source website analytics tool that does not rely on external services. Plausible allows you to track visitors, demographic data, device data, and much more. Plausible has a graphical interface that provides charts and maps that provide insight into the performance of your website server. Setting Plausible up on Akamai Connected Cloud and integrating it within your application is straightforward. +[Plausible](https://plausible.io/) is a free and open source website analytics tool that does not rely on external services. Plausible allows you to track visitors, demographic data, device data, and much more. Plausible has a graphical interface that provides charts and maps that provide insight into the performance of your website server. Setting Plausible up on Akamai Cloud and integrating it within your application is straightforward. ## Before You Begin @@ -152,4 +152,4 @@ Adding an SSL certificate to any website is an important step for both security ​ You're done! You can now create your account, add your website, and retrieve the code snippet you will include on your website. When adding the code to your website, make sure it's in a location that is visible on every page. This can be done with a plugin in WordPress or with the code injection on Ghost. The inclusion process will be slightly different for every platform and service. ​ -There are many more configuration options and changes you can make. For example, such as [adding a variable to disable registration](https://plausible.io/docs/self-hosting-configuration). Additionally, setting up email servers and notification was not covered in this article. For more options and advanced configuration, please check out [Plausible's official documentation](https://plausible.io/docs/self-hosting). \ No newline at end of file +There are many more configuration options and changes you can make. For example, such as [adding a variable to disable registration](https://plausible.io/docs/self-hosting-configuration). Additionally, setting up email servers and notification was not covered in this article. For more options and advanced configuration, please check out [Plausible's official documentation](https://plausible.io/docs/self-hosting). diff --git a/docs/guides/uptime/monitoring/monitoring-software/index.md b/docs/guides/uptime/monitoring/monitoring-software/index.md index f2eeb0ae412..0f4f172a988 100644 --- a/docs/guides/uptime/monitoring/monitoring-software/index.md +++ b/docs/guides/uptime/monitoring/monitoring-software/index.md @@ -88,7 +88,7 @@ Here are six open source system monitoring tools, two enterprise-grade commercia **[Nagios](https://www.nagios.com/)** is among the most venerable and widely used system monitoring applications. It’s a highly extensible monitoring and alerting system that can email or text you as soon as a system or service goes offline. -Nagios has two versions. Nagios Core is the open source version, while Nagios XI offers a proprietary interface and commercial support (which is also free for up to seven nodes). Akamai Connected Cloud offers [instructions for monitoring Debian and Ubuntu systems using Nagios](/docs/guides/monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004/). +Nagios has two versions. Nagios Core is the open source version, while Nagios XI offers a proprietary interface and commercial support (which is also free for up to seven nodes). Akamai Cloud offers [instructions for monitoring Debian and Ubuntu systems using Nagios](/docs/guides/monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004/). Both Nagios versions have flexible options for monitoring and alerting. Both versions also support a vast collection of "plugins" that extend monitoring capabilities and the types of monitored components. @@ -224,4 +224,4 @@ RMM products are strongest in situations when it comes to endpoint monitoring *a There are many options when it comes to system monitoring software. In fact, your best choice may be a mix of two or more tools. A useful starting point is to envision how you’d like to consume monitored data, then work backwards from there. This approach is far more efficient than configuring every single system and service in your organization, only to discover a given monitoring interface really doesn’t serve your needs after all. -Fortunately, with VM and container hosting on Akamai Connected Cloud, you can spin up a prototype monitoring setup to test out several options. [Contact Akamai](https://www.linode.com/company/contact/) today if you’d like to learn more about options for hosting system monitoring software. \ No newline at end of file +Fortunately, with VM and container hosting on Akamai Cloud, you can spin up a prototype monitoring setup to test out several options. [Contact Akamai](https://www.linode.com/company/contact/) today if you’d like to learn more about options for hosting system monitoring software. diff --git a/docs/products/platform/get-started/guides/beta-for-new-data-centers/index.md b/docs/products/platform/get-started/guides/beta-for-new-data-centers/index.md index a1978688265..c187104728d 100644 --- a/docs/products/platform/get-started/guides/beta-for-new-data-centers/index.md +++ b/docs/products/platform/get-started/guides/beta-for-new-data-centers/index.md @@ -124,4 +124,4 @@ Lish and Glish provide direct access to your Compute Instances, bypassing the ne ``` {{< /note >}} -- **Weblish/Glish Gateway:** `au-mel.webconsole.linode.com` \ No newline at end of file +- **Weblish/Glish Gateway:** `au-mel.webconsole.linode.com` diff --git a/docs/reference-architecture/auto-scaling-prometheus/_index.md b/docs/reference-architecture/auto-scaling-prometheus/_index.md index ab6877fc717..2380fa17b78 100644 --- a/docs/reference-architecture/auto-scaling-prometheus/_index.md +++ b/docs/reference-architecture/auto-scaling-prometheus/_index.md @@ -13,7 +13,7 @@ tab_group_main: This abstract provides a concrete example of how to autoscale a generic, highly available application runtime running on Compute Instances using Prometheus and a Jenkins CI/CD pipeline. For an introduction to CI/CD pipelines and use cases see [Introduction to Continuous Integration and Continuous Delivery (CI/CD)](/docs/guides/introduction-ci-cd/). -Cloud-based highly available workloads often need to scale horizontally when faced with periods of high demand, which can include traffic bursts based on marketing campaigns, new product launches, industry-based cyclical usage patterns, or unanticipated demand. Regardless of the reason, having the flexibility to reduce your costs when traffic is low, while also having the capability to expand your workload capacity on-demand, can be critical to customer satisfaction, your reputation, and your bottom line. This reference architecture demonstrates how to scale your workloads up or down within Akamai Connected Cloud using Compute Instances hosting your application runtime. +Cloud-based highly available workloads often need to scale horizontally when faced with periods of high demand, which can include traffic bursts based on marketing campaigns, new product launches, industry-based cyclical usage patterns, or unanticipated demand. Regardless of the reason, having the flexibility to reduce your costs when traffic is low, while also having the capability to expand your workload capacity on-demand, can be critical to customer satisfaction, your reputation, and your bottom line. This reference architecture demonstrates how to scale your workloads up or down within Akamai Cloud using Compute Instances hosting your application runtime. ### Figure 1: A Common Highly Available Application Architecture