diff --git a/00_Introduction/README.md b/00_Introduction/README.md index 79eba7e..75f5730 100644 --- a/00_Introduction/README.md +++ b/00_Introduction/README.md @@ -1,21 +1,27 @@ -# Welcome to the Azure Cosmos DB + Azure OpenAI Node.js Developer Guide +# Welcome to the Azure DocumentDB + Azure OpenAI Node.js Developer Guide ## Pre-requisites - [Azure account and subscription](https://azure.microsoft.com/free/) with Owner permissions - [Node.js 20.11.0 LTS or newer](https://nodejs.org/) - [Visual Studio Code](https://code.visualstudio.com/download) -- [Docker Desktop](https://www.docker.com/products/docker-desktop/) with [WSL 2 backend (if on Windows)](https://learn.docker.com/desktop/wsl/) +- [Docker Desktop](https://www.docker.com/products/docker-desktop/) with [WSL 2 backend (if on Windows)](https://docs.docker.com/desktop/features/wsl/) - [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) - [Bicep CLI](https://learn.microsoft.com/azure/azure-resource-manager/bicep/install#install-manually) - [Powershell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell?view=powershell-7.3) ## Why use this guide? -The future of software involves combining AI and data services, also known as intelligent applications. This guide is for MongoDB developers looking to implement intelligent applications quickly while leveraging existing skills. The content will focus on the developer journey implementing an Azure-based AI-enabled GPT-based chat application that is augmented using data stored in vCore for Azure Cosmos DB for MongoDB while leveraging Azure OpenAI services. +The future of software involves combining AI and data services, also known as intelligent applications. +This guide is for MongoDB developers looking to implement intelligent applications quickly while leveraging existing skills. +The content will focus on the developer journey implementing an Azure-based AI-enabled GPT-based chat application that is augmented using data stored in Azure DocumentDB while leveraging Azure OpenAI services. ## Introduction -This guide will walks through the creating intelligent solutions that combines vCore-based Azure Cosmos DB for MongoDB vector search and document retrieval with Azure OpenAI services to build a chat bot experience. The guide includes labs that build and deploy a sample chat app using these technologies, with a focus on vCore-based Azure Cosmos DB for MongoDB, Vector Search, and Azure OpenAI using the Node.js runtime and the JavaScript programming language. For those new to using Azure OpenAI and Vector Search technologies, the guide includes explanations of the core concepts and techniques used when implementing these technologies. +This guide will walks through the creating intelligent solutions that combines Azure DocumentDB with vector search capabilities powered by DiskANN and document retrieval with Azure OpenAI services to build a chat bot experience. +The guide includes labs that build and deploy a sample chat app using these technologies, with a focus on Azure DocumentDB, vector search powered by DiskANN, and Azure OpenAI using the Node.js programming language. +For those new to using Azure OpenAI and Vector Search technologies, the guide includes explanations of the core concepts and techniques used when implementing these technologies. -> **Note:** This developer guide is targeted towards Node.js developers. If you are a Python developer, then you may be interested in the Python version here: [https://github.com/AzureCosmosDB/Azure-OpenAI-Python-Developer-Guide](https://github.com/AzureCosmosDB/Azure-OpenAI-Python-Developer-Guide) \ No newline at end of file +> **Note:** This developer guide is targeted towards Node.js developers. + +If you are a Python developer, then you may be interested in the Python version here: [https://github.com/AzureCosmosDB/Azure-OpenAI-Python-Developer-Guide](https://github.com/AzureCosmosDB/Azure-OpenAI-Python-Developer-Guide/vcore) diff --git a/01_Azure_Overview/README.md b/01_Azure_Overview/README.md index 2d4b034..e11bc51 100644 --- a/01_Azure_Overview/README.md +++ b/01_Azure_Overview/README.md @@ -1,91 +1,116 @@ # Azure Overview -Millions of customers worldwide trust the Azure platform, and there are over 90,000 Cloud Solution Providers (CSPs) partnered with Microsoft to add extra benefits and services to the Azure platform. By leveraging Azure, organizations can easily modernize their applications, expedite application development, and adapt application requirements to meet the demands of their users. This section provides an overview of Azure, its services, and recommendations on how to get started. +Millions of customers worldwide trust the Azure platform, and there are over 90,000 Cloud Solution Providers (CSPs) partnered with Microsoft to add extra benefits and services to the Azure platform. +By leveraging Azure, organizations can easily modernize their applications, expedite application development, and adapt application requirements to meet the demands of their users. +This section provides an overview of Azure, its services, and recommendations on how to get started. ## Advantages of choosing Azure -By offering solutions on Azure, ISVs can access one of the largest B2B markets in the world. Through the [Azure Partner Builder's Program](https://partner.microsoft.com/marketing/azure-isv-technology-partners), Microsoft assists ISVs with the tools and platform to offer their solutions for customers to evaluate, purchase, and deploy with just a few clicks of the mouse. +By offering solutions on Azure, ISVs can access one of the largest B2B markets in the world. +Through the [Azure Partner Builder's Program](https://partner.microsoft.com/marketing/azure-isv-technology-partners), Microsoft assists ISVs with the tools and platform to offer their solutions for customers to evaluate, purchase, and deploy with just a few clicks of the mouse. -Microsoft's development suite includes such tools as the various [Visual Studio](https://visualstudio.microsoft.com/) products, [Azure DevOps](https://dev.azure.com/), [GitHub](https://github.com/), and low-code [Power Apps](https://powerapps.microsoft.com/). All of these contribute to Azure's success and growth through their tight integrations with the Azure platform. Organizations that adopt modern tools are 65% more innovative, according to a [2020 McKinsey & Company report.](https://azure.microsoft.com/mediahandler/files/resourcefiles/developer-velocity-how-software-excellence-fuels-business-performance/Developer-Velocity-How-software-excellence-fuels-business-performance-v4.pdf) +Microsoft's development suite includes such tools as the various [Visual Studio](https://visualstudio.microsoft.com/) products, [Azure DevOps](https://dev.azure.com/), [GitHub](https://github.com/), and low-code [Power Apps](https://powerapps.microsoft.com/). +All of these contribute to Azure's success and growth through their tight integrations with the Azure platform. +Organizations that adopt modern tools are 65% more innovative, according to a [2020 McKinsey & Company report](https://azure.microsoft.com/mediahandler/files/resourcefiles/developer-velocity-how-software-excellence-fuels-business-performance/Developer-Velocity-How-software-excellence-fuels-business-performance-v4.pdf). ![This image demonstrates common development tools on the Microsoft cloud platform to expedite application development.](media/ISV-Tech-Builders-tools-white.png "Microsoft cloud tooling") -To facilitate developers' adoption of Azure, Microsoft offers a [free subscription](https://azure.microsoft.com/free/search/) with $200 credit, applicable for thirty days; year-long access to free quotas for popular services and access to always free Azure service tiers. +To facilitate developers' adoption of Azure, Microsoft offers a [free Azure Account](https://azure.microsoft.com/free/search/) with $200 credit, applicable for thirty days; year-long access to free quotas for popular services and access to always free Azure service tiers. ## Introduction to Azure resource management -The [Azure Fundamentals Microsoft Learn Module](https://learn.microsoft.com/learn/modules/intro-to-azure-fundamentals/) demonstrates the different classifications of Azure Services. Moreover, Azure supports a variety of common tools, such as Visual Studio, PowerShell, and the Azure CLI, to manage Azure environments. +The [Azure Fundamentals Microsoft Learn Module](https://learn.microsoft.com/learn/modules/intro-to-azure-fundamentals/) demonstrates the different classifications of Azure Services. +Moreover, Azure supports a variety of common tools, such as Visual Studio, PowerShell, and the Azure CLI, to manage Azure environments. - ![IaaS and PaaS Azure service classification and categories](media/azure-services.png "Categories of Azure services") +![IaaS and PaaS Azure service classification and categories](media/azure-services.png "Categories of Azure services") ### The Azure resource management hierarchy -Azure provides a flexible resource hierarchy to simplify cost management and security. This hierarchy consists of four levels: +Azure provides a flexible resource hierarchy to simplify cost management and security. +This hierarchy consists of four levels: - **[Management groups](https://learn.microsoft.com/azure/governance/management-groups/overview)**: Management groups consolidate multiple Azure subscriptions for compliance and security purposes. - **Subscriptions**: Subscriptions govern cost control and access management. Azure users cannot provision Azure resources without a subscription. -- **[Resource groups](https://learn.microsoft.com/azure/azure-resource-manager/management/manage-resource-groups-portal)**: Resource groups consolidate the individual Azure resources for a given deployment. All provisioned Azure resources belong to one resource group. In this guide, it will be required to provision a *resource group* in an *subscription* to hold the required resources. +- **[Resource groups](https://learn.microsoft.com/azure/azure-resource-manager/management/manage-resource-groups-portal)**: Resource groups consolidate the individual Azure resources for a given deployment. All provisioned Azure resources belong to one resource group. In this guide, it will be required to provision a _resource group_ in a _subscription_ to hold the required resources. - Resource groups are placed in a geographic location that determines where metadata about that resource group is stored. - **Resources**: An Azure resource is an instance of a service. An Azure resource belongs to one resource group located in one subscription. + - Most Azure resources are provisioned in a particular region. ![This image shows Azure resource scopes.](media/scope-levels.png "Azure resource scopes") ### Create landing zone -An [Azure landing zone](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/landing-zone/) is the target environment defined as the final resting place of a cloud migration project. In most projects, the landing zone should be scripted via ARM templates for its initial setup. Finally, it should be customized with PowerShell or the Azure Portal to fit the workload's needs. First-time Azure users will find creating and deploying to DEV and TEST environments easy. +An [Azure landing zone](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/landing-zone/) is the target environment defined as the final resting place of a cloud migration project. +In most projects, the landing zone should be scripted via ARM templates for its initial setup. +Finally, it should be customized with PowerShell or the Azure Portal to fit the workload's needs. First-time Azure users will find creating and deploying to DEV and TEST environments easy. -To help organizations quickly move to Azure, Microsoft provides the Azure landing zone accelerator, which generates a landing zone ARM template according to an organization's core needs, governance requirements, and automation setup. The landing zone accelerator is available in the Azure portal. +To help organizations quickly move to Azure, Microsoft provides the Azure landing zone accelerator, which generates a landing zone ARM template according to an organization's core needs, governance requirements, and automation setup. +The landing zone accelerator is available in the Azure portal. ![This image demonstrates the Azure landing zone accelerator in the Azure portal, and how organizations can optimize Azure for their needs and innovate.](media/landing-zone-accelerator.png "Azure landing zone accelerator screenshot") ### Automating and managing Azure services -When it comes to managing Azure resources, there are many potential options. [Azure Resource Manager](https://learn.microsoft.com/azure/azure-resource-manager/management/overview) is the deployment and management service for Azure. It provides a management layer that enables users to create, update, and delete resources in Azure subscriptions. Use management features, like access control, locks, and tags, to secure and organize resources after deployment. +When it comes to managing Azure resources, there are many potential options. [Azure Resource Manager](https://learn.microsoft.com/azure/azure-resource-manager/management/overview) is the deployment and management service for Azure. +It provides a management layer that enables users to create, update, and delete resources in Azure subscriptions. +Use management features, like access control, locks, and tags, to secure and organize resources after deployment. All Azure management tools, including the [Azure CLI](https://learn.microsoft.com/cli/azure/what-is-azure-cli), [Azure PowerShell](https://learn.microsoft.com/powershell/azure/what-is-azure-powershell?view=azps-7.1.0) module, [Azure REST API](https://learn.microsoft.com/rest/api/azure/), and browser-based Portal, interact with the Azure Resource Manager layer and [Identity and access management (IAM)](https://learn.microsoft.com/azure/role-based-access-control/overview) security controls. - ![This image demonstrates how the Azure Resource Manager provides a robust, secure interface to Azure resources.](media/consistent-management-layer.png "Azure Resource Manager explained") +![This image demonstrates how the Azure Resource Manager provides a robust, secure interface to Azure resources.](media/consistent-management-layer.png "Azure Resource Manager explained") -Access control to all Azure services is offered via the [Azure role-based access control (Azure RBAC)](https://learn.microsoft.com/azure/role-based-access-control/overview) natively built into the management platform. Azure RBAC is a system that provides fine-grained access management of Azure resources. Using Azure RBAC, it is possible to segregate duties within teams and grant only the amount of access to users that they need to perform their jobs. +Access control to all Azure services is offered via the [Azure role-based access control (Azure RBAC)](https://learn.microsoft.com/azure/role-based-access-control/overview) natively built into the management platform. +Azure RBAC is a system that provides fine-grained access management of Azure resources. +Using Azure RBAC, it is possible to segregate duties within teams and grant only the amount of access to users that they need to perform their jobs. ### Azure management tools -The flexibility and variety of Azure's management tools make it intuitive for any user, irrespective of their skill level with specific technologies. As an individual's skill level and administration needs mature, Azure has the right tools to match those needs. +The flexibility and variety of Azure's management tools make it intuitive for any user, irrespective of their skill level with specific technologies. +As an individual's skill level and administration needs mature, Azure has the right tools to match those needs. ![Azure service management tool maturity progression.](media/azure-management-tool-maturity.png "Azure service management tool") #### Azure portal -As a new Azure user, the first resource a person will be exposed to is the Azure Portal. The **Azure Portal** gives developers and architects a view of the state of their Azure resources. It supports extensive user configuration and simplifies reporting. The **[Azure mobile app](https://azure.microsoft.com/get-started/azure-portal/mobile-app/)** provides similar features for users that are away from their main desktop or laptop. +As a new Azure user, the first resource a person will be exposed to is the Azure Portal. The **Azure Portal** gives developers and architects a view of the state of their Azure resources. +It supports extensive user configuration and simplifies reporting. +The **[Azure mobile app](https://azure.microsoft.com/get-started/azure-portal/mobile-app/)** provides similar features for users that are away from their main desktop or laptop. - ![The picture shows the initial Azure service list.](media/azure-portal-services.png "Azure portal Services") +![The picture shows the initial Azure service list.](media/azure-portal-services.png "Azure portal Services") Azure runs on a common framework of backend resource services, and every action taken in the Azure portal translates into a call to a set of backend APIs developed by the respective engineering team to read, create, modify, or delete resources. ##### Azure Marketplace -[Azure Marketplace](https://learn.microsoft.com/marketplace/azure-marketplace-overview) is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. In Azure Marketplace, it is possible to find, try, buy, and deploy the software and services needed to build new solutions and manage the cloud infrastructure. The catalog includes solutions for different industries and technical areas, free trials, and consulting services from Microsoft partners. +[Azure Marketplace](https://learn.microsoft.com/marketplace/azure-marketplace-overview) is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. +In Azure Marketplace, it is possible to find, try, buy, and deploy the software and services needed to build new solutions and manage the cloud infrastructure. +The catalog includes solutions for different industries and technical areas, free trials, and consulting services from Microsoft partners. ![The picture shows an example of Azure Marketplace search results.](media/azure-marketplace-search-results.png "Azure Marketplace Results") ##### Evolving -Moving workloads to Azure alleviates some administrative burdens, but not all. Even though there is no need to worry about the data center, there is still a responsibility for service configuration and user access. Applications will need resource authorization. +Moving workloads to Azure alleviates some administrative burdens, but not all. +Even though there is no need to worry about the data center, there is still a responsibility for service configuration and user access. +Applications will need resource authorization. Using the existing command-line tools and REST APIs, it is possible to build custom tools to automate and report resource configurations that do not meet organizational requirements. #### Azure PowerShell and CLI -**Azure PowerShell** and the **Azure CLI** (for Bash shell users) are useful for automating tasks that cannot be performed in the Azure portal. Both tools follow an *imperative* approach, meaning that users must explicitly script the creation of resources in the correct order. +**Azure PowerShell** and the **Azure CLI** (for Bash shell users) are useful for automating tasks that cannot be performed in the Azure portal. +Both tools follow an _imperative_ approach, meaning that users must explicitly script the creation of resources in the correct order. - ![Shows an example of the Azure CLI.](media/azure-cli-example.png "Azure CLI Example") +![Shows an example of the Azure CLI.](media/azure-cli-example.png "Azure CLI Example") -There are subtle differences between how each of these tools operates and the actions that can be accomplished. Use the [Azure command-line tool guide](https://learn.microsoft.com/azure/developer/azure-cli/choose-the-right-azure-command-line-tool) to determine the right tool to meet the target goal. +There are subtle differences between how each of these tools operates and the actions that can be accomplished. +Use the [Azure command-line tool guide](https://learn.microsoft.com/azure/developer/azure-cli/choose-the-right-azure-command-line-tool) to determine the right tool to meet the target goal. #### Azure CLI -It is possible to run the Azure CLI and Azure PowerShell from the [Azure Cloud Shell](https://shell.azure.com), but it does have some limitations. It is also possible to run these tools locally. +It is possible to run the Azure CLI and Azure PowerShell from the [Azure Cloud Shell](https://shell.azure.com), but it does have some limitations. +It is also possible to run these tools locally. To use the Azure CLI, [download the CLI tools from Microsoft.](https://learn.microsoft.com/cli/azure/install-azure-cli) @@ -93,13 +118,15 @@ To use the Azure PowerShell cmdlets, install the `Az` module from the PowerShell ##### Azure Cloud Shell -The Azure Cloud Shell provides Bash and PowerShell environments for managing Azure resources imperatively. It also includes standard development tools, like Visual Studio Code, and files are persisted in an Azure Files share. +The Azure Cloud Shell provides Bash and PowerShell environments for managing Azure resources imperatively. +It also includes standard development tools, like Visual Studio Code, and files are persisted in an Azure Files share. -Launch the Cloud Shell in a browser at [https://shell.azure.com](https://shell.azure.com). +Launch the Cloud Shell in a browser at [Azure Cloud Shell](https://shell.azure.com). #### PowerShell Module -The Azure portal and Windows PowerShell can be used for managing Azure Cosmos DB for Mongo DB API. To get started with Azure PowerShell, install the [Azure PowerShell cmdlets](https://learn.microsoft.com/powershell/module/az.cosmosdb/) for Cosmos DB with the following PowerShell command in an administrator-level PowerShell window: +The Azure portal and Windows PowerShell can be used for managing Azure DocumentDB. +To get started with Azure PowerShell, install the [Azure PowerShell cmdlets](https://learn.microsoft.com/powershell/module/az.cosmosdb/) for Azure Cosmos DB with the following PowerShell command in an administrator-level PowerShell window: ```PowerShell Install-Module -Name Az.CosmosDB @@ -107,19 +134,31 @@ Install-Module -Name Az.CosmosDB #### Infrastructure as Code -[Infrastructure as Code (IaC)](https://learn.microsoft.com/devops/deliver/what-is-infrastructure-as-code) provides a way to describe or declare what infrastructure looks like using descriptive code. The infrastructure code is the desired state. The environment will be built when the code runs and completes. One of the main benefits of IaC is that it is human readable. Once the environment code is proven and tested, it can be versioned and saved into source code control. Developers can review the environment changes over time. +[Infrastructure as Code (IaC)](https://learn.microsoft.com/devops/deliver/what-is-infrastructure-as-code) provides a way to describe or declare what infrastructure looks like using descriptive code. +The infrastructure code is the desired state. +The environment will be built when the code runs and completes. +One of the main benefits of IaC is that it is human readable. +Once the environment code is proven and tested, it can be versioned and saved into source code control. +Developers can review the environment changes over time. -There are a few options of IaC tooling to choose from when provisioning and managing Azure resources. These include Azure-native tools from Microsoft, like ARM templates and Azure Bicep, as well as third-party tools popular within the industry like HashiCorp Terraform. +There are a few options of IaC tooling to choose from when provisioning and managing Azure resources. +These include Azure-native tools from Microsoft, like ARM templates and Azure Bicep, as well as third-party tools popular within the industry like HashiCorp Terraform. ##### ARM templates -[ARM templates](https://learn.microsoft.com/azure/azure-resource-manager/templates/) can deploy Azure resources in a *declarative* manner. Azure Resource Manager can potentially create the resources in an ARM template in parallel. ARM templates can be used to create multiple identical environments, such as development, staging, and production environments. +[ARM templates](https://learn.microsoft.com/azure/azure-resource-manager/templates/) can deploy Azure resources in a _declarative_ manner. +Azure Resource Manager can potentially create the resources in an ARM template in parallel. +ARM templates can be used to create multiple identical environments, such as development, staging, and production environments. - ![The picture shows an example of an ARM template JSON export.](media/azure-template-json-example.png "Azure Template JSON") +![The picture shows an example of an ARM template JSON export.](media/azure-template-json-example.png "Azure Template JSON") ##### Bicep -Reading, updating, and managing the ARM template JSON code can be difficult for a reasonably sized environment. What if there was a tool that translates simple declarative statements into ARM templates? Better yet, what if there was a tool that took existing ARM templates and translated them into a simple configuration? [Bicep](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview) is a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. Bicep files define the infrastructure to deploy to Azure and then use that file throughout the development lifecycle to repeatedly deploy infrastructure changes. Resources are deployed consistently. +Reading, updating, and managing the ARM template JSON code can be difficult for a reasonably sized environment. +What if there was a tool that translates simple declarative statements into ARM templates? +Better yet, what if there was a tool that took existing ARM templates and translated them into a simple configuration? [Bicep](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview) is a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. +Bicep files define the infrastructure to deploy to Azure and then use that file throughout the development lifecycle to repeatedly deploy infrastructure changes. +Resources are deployed consistently. By using the Azure CLI it is possible to decompile ARM templates to Bicep using the following: @@ -127,15 +166,18 @@ By using the Azure CLI it is possible to decompile ARM templates to Bicep using az bicep decompile --file template.json ``` -Additionally, the [Bicep playground](https://aka.ms/bicepdemo) tool can perform similar decompilation of ARM templates. +Additionally, the [Bicep Playground](https://azure.github.io/bicep/) tool can perform similar decompilation of ARM templates. -![Sample Bicep code that deploys Azure Cosmos DB for MongoDB](media/bicep_code.png) +![Sample Bicep code that deploys Azure DocumentDB](media/bicep_code.png) ##### Terraform -[Hashicorp Terraform](https://www.terraform.io/) is an open-source tool for provisioning and managing cloud infrastructure resources. [Terraform](https://learn.microsoft.com/azure/developer/terraform/overview) simplifies the deployment of Azure services, including Azure Kubernetes Service, Azure Cosmos DB, and Azure AI, through infrastructure-as-code to automate provisioning and management of Azure services. Terraform is also adept at deploying infrastructure across multiple cloud providers. It enables developers to use consistent tooling to manage each infrastructure definition. +[Hashicorp Terraform](https://www.terraform.io/) is an open-source tool for provisioning and managing cloud infrastructure resources. +[Terraform](https://learn.microsoft.com/azure/developer/terraform/overview) simplifies the deployment of Azure services, including Azure Kubernetes Service, Azure Cosmos DB, and Azure AI, through infrastructure-as-code to automate provisioning and management of Azure services. +Terraform is also adept at deploying infrastructure across multiple cloud providers. +It enables developers to use consistent tooling to manage each infrastructure definition. -![Sample Terraform code that deploys Azure Cosmos DB for MongoDB](media/terraform_code.png) +![Sample Terraform code that deploys Azure DocumentDB](media/terraform_code.png) #### Other tips @@ -145,6 +187,7 @@ Here are some best practices to follow for Azure deployments. - **Utilize Management Groups** Create at least three levels of management groups. -- **Adopt a naming convention:** Azure resource names should include business details, such as the organization department, and operational details for IT personnel, like the workload. Defining an [Azure resource naming convention](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming) will help the organization standardize on a common naming convention that will help better identify resources once created. +- **Adopt a naming convention:** Azure resource names should include business details, such as the organization department, and operational details for IT personnel, like the workload. +Defining an [Azure resource naming convention](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming) will help the organization standardize on a common naming convention that will help better identify resources once created. - **Adopt other Azure governance tools:** Azure provides mechanisms such as [resource tags](https://learn.microsoft.com/azure/azure-resource-manager/management/tag-resources?tabs=json) and [resource locks](https://learn.microsoft.com/azure/azure-resource-manager/management/lock-resources?tabs=json) to facilitate compliance, cost management, and security. diff --git a/02_Overview_Cosmos_DB/README.md b/02_Overview_Cosmos_DB/README.md index 88cd58a..b7e382b 100644 --- a/02_Overview_Cosmos_DB/README.md +++ b/02_Overview_Cosmos_DB/README.md @@ -1,27 +1,33 @@ # Overview of Azure Cosmos DB -[Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/introduction) is a globally distributed, multi-model database service that enables querying and storing data using NoSQL models using one of five APIs: SQL (document database), Cassandra (column-family), MongoDB (document database), Azure Table, and graph database. It provides turnkey global distribution, elastic scaling of throughput and storage worldwide, single-digit millisecond latencies at the 99th percentile, and guaranteed high availability with multi-homing capabilities. Azure Cosmos DB provides comprehensive service level agreements (SLAs) for throughput, latency, availability, and consistency guarantees, something not found in any other database service. +[Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/introduction) is a globally distributed, multi-model database service for both NoSQL and relational workloads. +It supports multiple APIs: NoSQL, MongoDB, PostgreSQL, Cassandra, Gremlin, and Table—covering document, relational, column-family, graph, and key-value data models. +The service offers turnkey global distribution with elastic scaling of throughput and storage. +It delivers single-digit millisecond latencies at the 99th percentile and guarantees high availability through multi-homing capabilities. +Azure Cosmos DB provides comprehensive service level agreements (SLAs) covering throughput, latency, availability, and consistency—a unique combination among cloud database services. ## Azure Cosmos DB and AI -The surge of AI-powered applications has led to the need to integrate data from multiple data stores, introducing another layer of complexity as each data store tends to have its own workflow and operational performance. Azure Cosmos DB simplifies this process by providing a unified platform for all data types, including AI data. Azure Cosmos DB supports relational, document, vector, key-value, graph, and table data models, making it an ideal platform for AI applications. The wide array of data model support combined with guaranteed high availability, high throughput, low latency, and tunable consistency are huge advantages when building these types of applications. +The surge of AI-powered applications has led to the need to integrate data from multiple data stores, introducing another layer of complexity as each data store tends to have its own workflow and operational performance. +Azure Cosmos DB simplifies this process by providing a unified platform for all data types, including AI data. +Azure Cosmos DB supports relational, document, vector, key-value, graph, and table data models, making it an ideal platform for AI applications. +The wide array of data model support combined with guaranteed high availability, high throughput, low latency, and tunable consistency are huge advantages when building these types of applications. -## Azure Cosmos DB for Mongo DB +## Azure DocumentDB -The focus for this developer guide is [Azure Cosmos DB for MongoDB](https://learn.microsoft.com/azure/cosmos-db/mongodb/introduction). Developers can leverage their current MongoDB expertise and use their preferred MongoDB drivers, SDKs, and tools simply by directing applications to the connection string for on the Azure Cosmos DB for MongoDB account. +The focus for this developer guide is [Azure DocumentDB](https://learn.microsoft.com/azure/documentdb/overview). +Azure DocumentDB (previously known as vCore-based Azure Cosmos DB for MongoDB) is a fully managed MongoDB-compatible database service optimized for modern application development. +Developers can apply their existing MongoDB expertise and continue using their preferred MongoDB drivers, SDKs, and tools by simply pointing applications to the Azure DocumentDB connection string. -### Azure Cosmos DB for Mongo DB API Architectures +### Azure DocumentDB Architecture -The [RU architecture](https://learn.microsoft.com/azure/cosmos-db/mongodb/ru/introduction) for Azure Cosmos DB for MongoDB offers instantaneous scalability with zero warmup period, automatic and transparent sharding, and 99.999% availability. It supports active-active databases across multiple regions, cost-efficient, granular, unlimited scalability, real-time analytics, and serverless deployments paying only per operation. +Azure DocumentDB is powered by the open-source [DocumentDB engine](https://github.com/documentdb/documentdb), which is built on PostgreSQL and provides full MongoDB wire protocol compatibility. +This open-source foundation, released under the permissive MIT license, gives developers complete transparency and flexibility. -[vCore-based Azure Cosmos DB for MongoDB architecture](https://learn.microsoft.com/azure/cosmos-db/mongodb/vcore/introduction) integrates AI-based applications with private organizational data, with text indexing for easy querying. Simplify the development process with high-capacity vertical scaling and free 35-day backups with a point-in-time restore (PITR). +The service provides flexible and scalable data management with a schema-agnostic design. +It supports both vertical and horizontal scaling to handle high-capacity workloads, with no shard key required until your database surpasses terabytes. +You can shard existing databases automatically with no downtime and scale clusters up or down without interrupting your applications. +For more information, see [Azure DocumentDB scalability and architecture](https://learn.microsoft.com/azure/documentdb/scalability-overview). -The [choice between vCore and Request Units (RU)](https://learn.microsoft.com/azure/cosmos-db/mongodb/choose-model) in Azure Cosmos DB for MongoDB API depends on the workload. A list of [compatibility and feature support between RU and vCore](https://learn.microsoft.com/azure/cosmos-db/mongodb/vcore/compatibility) is available. - -vCore provides predictable performance and cost and is ideal for running high-performance, mission-critical workloads with low latency and high throughput. With vCore, the number of vCPUs and the memory the database needs is configurable and can be scaled up or down as needed. - -Conversely, RU is a consumption-based model that charges based on the number of operations the database performs, including reads, writes, and queries. RU is ideal for scenarios where the workload has unpredictable traffic patterns or a need to optimize cost for bursty workloads. - -A steady-state workload with predictable traffic patterns is best suited for vCore since it provides more predictable performance and cost. However, RU may be a better choice if the workload has unpredictable traffic patterns or requires bursty performance since it allows for paying only for the resources used. - ->**NOTE**: AI-supporting workloads, such as vector search, must use the vCore architecture, as vector search is not supported with RU accounts. +Azure DocumentDB includes an [integrated vector database](https://learn.microsoft.com/azure/documentdb/vector-search) that enables you to store, index, and query high-dimensional vector data alongside your original data. +This eliminates the need to replicate data in a separate vector database, reducing cost and complexity while enabling AI-powered applications such as semantic search, recommendations, and retrieval-augmented generation (RAG). diff --git a/03_Overview_Azure_OpenAI/README.md b/03_Overview_Azure_OpenAI/README.md index 9683227..a187cf8 100644 --- a/03_Overview_Azure_OpenAI/README.md +++ b/03_Overview_Azure_OpenAI/README.md @@ -1,6 +1,8 @@ -# Overview of Azure OpenAI +# Overview of Azure OpenAI and Foundry Tools -Azure OpenAI is a collaboration between Microsoft Azure and OpenAI, a leading research organization in artificial intelligence. It is a cloud-based platform that enables developers and data scientists to build and deploy AI models quickly and easily. With Azure OpenAI, users can access a wide range of AI tools and technologies to create intelligent applications, including natural language processing, computer vision, and deep learning. +Azure OpenAI is a collaboration between Microsoft Azure and OpenAI, a leading research organization in artificial intelligence. +It is a cloud-based platform that enables developers and data scientists to build and deploy AI models quickly and easily. +With Azure OpenAI, users can access a wide range of AI tools and technologies to create intelligent applications, including natural language processing, computer vision, and deep learning. Azure OpenAI is designed to accelerate the development of AI applications, allowing users to focus on creating innovative solutions that deliver value to their organizations and customers. @@ -11,18 +13,23 @@ Here are ways that Azure OpenAI can help developers: - **Customization** - Developers can also fine-tune the included pre-trained models with their own data with minimal coding, providing an opportunity to create more personalized and specialized AI applications. - **Documentation and resources** - Azure OpenAI provides comprehensive documentation and resources to help developers get started quickly. - **Scalability and reliability** - Hosted on Microsoft Azure, the OpenAI service provides robust scalability and reliability that developers can leverage to deploy their applications. -- **Responsible AI** - Azure OpenAI promotes responsible AI by adhering to ethical principles, providing explainability tools, governance features, diversity and inclusion support, and collaboration opportunities. These measures help ensure that AI models are unbiased, explainable, trustworthy, and used in a responsible and compliance manner. +- **Responsible AI** - Azure OpenAI promotes responsible AI by adhering to ethical principles, providing explainability tools, governance features, diversity and inclusion support, and collaboration opportunities. +These measures help ensure that AI models are unbiased, explainable, trustworthy, and used in a responsible and compliance manner. - **Community support** - With an active developer community developers can seek help via forums and other community support channels. ## Comparison of Azure OpenAI and OpenAI -Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-4, GPT-3, Codex, DALL-E, and Whisper models with the security and enterprise promise of Azure. Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other. +Azure OpenAI Service gives customers advanced language AI with OpenAI GPT-4, GPT-3, Codex, DALL-E, and Whisper models with the security and enterprise promise of Azure. +Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other. -With Azure OpenAI, customers get the security capabilities of Microsoft Azure while running the same models as OpenAI. Azure OpenAI offers private networking, regional availability, and responsible AI content filtering. +With Azure OpenAI, customers get the security capabilities of Microsoft Azure while running the same models as OpenAI. +Azure OpenAI offers private networking, regional availability, and responsible AI content filtering. ## Azure OpenAI Data Privacy and Security -Azure OpenAI stores and processes data to provide the service and to monitor for uses that violate the applicable product terms. Azure OpenAI is fully controlled by Microsoft. Microsoft hosts the OpenAI models in Microsoft Azure for the usage of Azure OpenAI, and does not interact with any services operated by OpenAI. +Azure OpenAI stores and processes data to provide the service and to monitor for uses that violate the applicable product terms. +Azure OpenAI is fully controlled by Microsoft. +Microsoft hosts the OpenAI models in Microsoft Azure for the usage of Azure OpenAI, and does not interact with any services operated by OpenAI. Here are a few important things to know in regards to the security and privacy of prompts (inputs) and completions (outputs), embeddings, and training data when using Azure OpenAI: @@ -35,20 +42,24 @@ Here are a few important things to know in regards to the security and privacy o ## Azure AI Platform -Developers can use the power of AI, cloud-scale data, and cloud-native app development to create highly differentiated digital experiences and establish leadership among competitors. Build or modernize intelligent applications that take advantage of industry-leading AI technology and leverage real-time data and analytics to deliver adaptive, responsive, and personalized experiences. +Developers can use the power of AI, cloud-scale data, and cloud-native app development to create highly differentiated digital experiences and establish leadership among competitors. +Build or modernize intelligent applications that take advantage of industry-leading AI technology and leverage real-time data and analytics to deliver adaptive, responsive, and personalized experiences. -The Azure platform of managed AI, containers, and database services, along with offerings developed by or in partnership with key software vendors, enables developers to build, deploy, and scale applications with speed, flexibility, and enterprise-grade security. This platform has been used by market leaders like The NBA, H&R Block, Real Madrid Football Club, Bosch, and Nuance to develop their own intelligent apps. +The Azure platform of managed AI, containers, and database services, along with offerings developed by or in partnership with key software vendors, enables developers to build, deploy, and scale applications with speed, flexibility, and enterprise-grade security. +This platform has been used by market leaders like The NBA, H&R Block, Real Madrid Football Club, Bosch, and Nuance to develop their own intelligent apps. -Developers can use Azure AI Services, along with other Azure services, to build and modernize intelligent apps on Azure. Examples of this could be: +Developers can use Foundry Tools, along with other Azure services, to build and modernize intelligent apps on Azure. +Examples of this could be: -- Build new with Azure Kubernetes Service or Azure Container Apps, Azure Cosmos DB, and Azure AI Services -- Modernize with Azure Kubernetes Service, Azure SQL or Azure Database for PostgresSQL, and Azure AI Services +- Build new with Azure Kubernetes Service or Azure Container Apps, Azure Cosmos DB, and Foundry Tools +- Modernize with Azure Kubernetes Service, Azure SQL or Azure Database for PostgresSQL, and Foundry Tools -### Azure AI Services +### Foundry Tools -While this guide focuses on building intelligent apps using Azure OpenAI combined with vCore-based Azure Cosmos DB for MongoDB, the Azure AI Platform consists of many additional AI services. Each AI service is built to fit a specific AI and/or Machine Learning (ML) need. +While this guide focuses on building intelligent apps using Azure OpenAI combined with Azure DocumentDB, the Azure AI Platform consists of many additional AI services. +Each AI service is built to fit a specific AI and/or Machine Learning (ML) need. -Here's a list of the AI services within the [Azure AI platform](https://learn.microsoft.com/azure/ai-services/what-are-ai-services): +Here's a list of [Foundry Tools](https://learn.microsoft.com/azure/ai-services/what-are-ai-services) within Microsoft Foundry: | Service | Description | | --- | --- | @@ -67,24 +78,31 @@ Here's a list of the AI services within the [Azure AI platform](https://learn.mi | Video Indexer | Extract actionable insights from videos | | Vision | Analyze content in images and videos | -> **Note:** Follow this link for additional tips to help in determining the which Azure AI service is most appropriate for a specific project requirement: +> **Note:** Follow this link for additional tips to help in determining the which Foundry Tools is most appropriate for a specific project requirement: [Azure AI products](https://azure.microsoft.com/products/category/ai) -The tools that used to customize and configure models are different from those used to call the Azure AI services. Out of the box, most Azure AI services allow for sending data and receive insights without any customization. +The tools that used to customize and configure models are different from those used to call the Foundry Tools. +Out of the box, most Foundry Tools allow for sending data and receive insights without any customization. For example: - Sending an image to the Azure AI Vision service to detect words and phrases or count the number of people in the frame - Sending an audio file to the Speech service and get transcriptions and translate the speech to text at the same time -Azure offers a wide range of tools that are designed for different types of users, many of which can be used with Azure AI services. Designer-driven tools are the easiest to use, and are quick to set up and automate, but might have limitations when it comes to customization. The REST APIs and client libraries provide users with more control and flexibility, but require more effort, time, and expertise to build a solution. When using REST APIs and client libraries, there is an expectation that the developer is comfortable working with modern programming languages like C#, Java, Python, JavaScript, or another popular programming language. +Azure offers a wide range of tools that are designed for different types of users, many of which can be used with Foundry Tools. +Designer-driven tools are the easiest to use, and are quick to set up and automate, but might have limitations when it comes to customization. +The REST APIs and client libraries provide users with more control and flexibility, but require more effort, time, and expertise to build a solution. +When using REST APIs and client libraries, there is an expectation that the developer is comfortable working with modern programming languages like C#, Java, Python, JavaScript, or another popular programming language. ### Azure Machine Learning -[Azure Machine Learning](https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning?view=azureml-api-2) is a cloud service for accelerating and managing the machine learning (ML) project lifecycle. ML professionals, data scientists, and engineers can use it in their day-to-day workflows to train and deploy models and manage machine learning operations (MLOps). +[Azure Machine Learning](https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning?view=azureml-api-2) is a cloud service for accelerating and managing the machine learning (ML) project lifecycle. +ML professionals, data scientists, and engineers can use it in their day-to-day workflows to train and deploy models and manage machine learning operations (MLOps). -Azure Machine Learning can be used to create a model or use a model built from an open-source platform, such as PyTorch, TensorFlow, or scikit-learn. Additionally, MLOps tools help monitor, retrain, and redeploy models. +Azure Machine Learning can be used to create a model or use a model built from an open-source platform, such as PyTorch, TensorFlow, or scikit-learn. +Additionally, MLOps tools help monitor, retrain, and redeploy models. -ML projects often require a team with a varied skill set to build and maintain. Azure Machine Learning has tools that help enable: +ML projects often require a team with a varied skill set to build and maintain. +Azure Machine Learning has tools that help enable: - Collaboration within a team via shared notebooks, compute resources, serverless compute, data, and environments @@ -94,19 +112,21 @@ ML projects often require a team with a varied skill set to build and maintain. - Running machine learning workloads anywhere with built-in governance, security, and compliance -Enterprises working in the Microsoft Azure cloud can use familiar security and role-based access control for infrastructure. A project can be set up to deny access to protected data and select operations. +Enterprises working in the Microsoft Azure cloud can use familiar security and role-based access control for infrastructure. +A project can be set up to deny access to protected data and select operations. #### Azure Machine Learning vs Azure Open AI -Many of the Azure AI services are suited to a very specific AI / ML need. The Azure Machine Learning and Azure OpenAI services offer more flexible usage based on the solution requirements. +Many of the Foundry Tools are suited to a very specific AI / ML need. +The Azure Machine Learning and Azure OpenAI services offer more flexible usage based on the solution requirements. Here are a couple differentiators to help determine which of these to services to use when comparing the two: - Azure Machine Learning service is appropriate for solutions where a custom model needs to be trained specifically on private data. -- Azure OpenAI service is appropriate for solutions that require pre-trained models that provide natural language processing or vision services, such as the GPT-4 or DALL-E models from OpenAI. +- Azure OpenAI service is appropriate for solutions that require pre-trained models that provide natural language processing or vision services, such as the GPT (gpt-5, gpt-5-mini), DALL-E (DALL-E 3), or Sora (Sora-2) models from OpenAI. -If the solution requires other more task specific AI features, then one of the other Azure AI services should be considered. +If the solution requires other more task specific AI features, then one of the other Foundry Tools should be considered. ### Azure AI Studio @@ -118,11 +138,11 @@ Specifically, Azure AI Studio combines: - The generative AI model deployment, testing, and custom data integration capabilities of Azure OpenAI service. -- Integration with Azure AI Services for speech, vision, language, document intelligence, and content safety. +- Integration with Foundry Tools for speech, vision, language, document intelligence, and content safety. Azure AI Studio enables teams to collaborate efficiently and effectively on AI projects, such as developing custom copilot applications that use large language models (LLMs). -![Azure AI Studio screenshot](images/2024-01-23-17-52-46.png) +![Microsoft AI Foundry screenshot](images/Microsoft_AI_Foundry_2025_11_30.png) Tasks accomplished using Azure AI Studio include: @@ -131,4 +151,4 @@ Tasks accomplished using Azure AI Studio include: - Integrating data from custom data sources to support a retrieval augmented generation (RAG) approach to prompt engineering for generative AI models. - Using prompt flow to define workflows that integrate models, prompts, and custom processing. - Integrating content safety filters into a generative AI solution to mitigate potential harms. -- Extending a generative AI solution with multiple AI capabilities using Azure AI services. +- Extending a generative AI solution with multiple AI capabilities using Foundry Tools. diff --git a/03_Overview_Azure_OpenAI/images/2024-01-23-17-52-46.png b/03_Overview_Azure_OpenAI/images/2024-01-23-17-52-46.png deleted file mode 100644 index 2c51e31..0000000 Binary files a/03_Overview_Azure_OpenAI/images/2024-01-23-17-52-46.png and /dev/null differ diff --git a/03_Overview_Azure_OpenAI/images/Microsoft_AI_Foundry_2025_11_30.png b/03_Overview_Azure_OpenAI/images/Microsoft_AI_Foundry_2025_11_30.png new file mode 100644 index 0000000..b515980 Binary files /dev/null and b/03_Overview_Azure_OpenAI/images/Microsoft_AI_Foundry_2025_11_30.png differ diff --git a/04_Overview_AI_Concepts/README.md b/04_Overview_AI_Concepts/README.md index d3c1636..99fff87 100644 --- a/04_Overview_AI_Concepts/README.md +++ b/04_Overview_AI_Concepts/README.md @@ -2,21 +2,29 @@ ## Large Language Models (LLM) -A Large Language Models (LLM) is a type of AI that can process and produce natural language text. LLMs are "general purpose" AI models trained using massive amounts of data gathered from various sources; like books, articles, webpages, and images to discover patterns and rules of language. +A Large Language Models (LLM) is a type of AI that can process and produce natural language text. +LLMs are "general purpose" AI models trained using massive amounts of data gathered from various sources; like books, articles, webpages, and images to discover patterns and rules of language. -LLMs are complex and built using a neural network architecture. They are trained using large amounts of information, and calculate millions of parameters. From a developer perspective, the APIs expose by Azure OpenAI Service enable the LLMs to be easily integrated into enterprise solutions without requiring knowledge of how to build to train the models. +LLMs are complex and built using a neural network architecture. +They are trained using large amounts of information, and calculate millions of parameters. +From a developer perspective, the APIs expose by Azure OpenAI Service enable the LLMs to be easily integrated into enterprise solutions without requiring knowledge of how to build to train the models. Understanding the capabilities of what an LLM can do is important when deciding to use it for a solution: -- **Understand language** - An LLM is a predictive engine that pulls patterns together based on pre-existing text to produce more text. It doesn't understand language or math. +- **Understand language** - An LLM is a predictive engine that pulls patterns together based on pre-existing text to produce more text. +It doesn't really understand language or math. - **Understand facts** - An LLM doesn't have separate modes for information retrieval and creative writing; it simply predicts the next most probable token. -- **Understand manners, emotion, or ethics** - An LLM can't exhibit anthropomorphism or understand ethics. The output of a foundational model is a combination of training data and prompts. +- **Understand manners, emotion, or ethics** - An LLM can't exhibit anthropomorphism or understand ethics. +The output of a foundational model is a combination of training data and prompts. ### Foundational Models -Foundational Models are specific instances or versions of an LLM. Examples of these would be GPT-3, GPT-4, or Codex. Foundational models are trained and fine-tuned on a large corpus of text, or code in the case of a Codex model instance. +Foundational Models are specific instances or versions of an LLM. +Examples of these would be GPT-3, GPT-4, or Codex. +Foundational models are trained and fine-tuned on a large corpus of text, or code in the case of a Codex model instance. -A foundational model takes in training data in all different formats and uses a transformer architecture to build a general model. Adaptions and specializations can be created to achieve certain tasks via prompts or fine-tuning. +A foundational model takes in training data in all different formats and uses a transformer architecture to build a general model. +Adaptions and specializations can be created to achieve certain tasks via prompts or fine-tuning. ### Difference between LLM and traditional Natural Language Processing (NLP) @@ -34,7 +42,10 @@ Here are a few things that separate NLPs from LLMs: ### What is a prompt? -A prompt is an input or instruction provided to an Artificial Intelligence (AI) model to direct its behavior and produce the desired results. The quality and specificity of the prompt are crucial in obtaining precise and relevant outputs. A well-designed prompt can ensure that the AI model generates the desired information or completes the intended task effectively. Some typical prompts include summarization, question answering, text classification, and code generation. +A prompt is an input or instruction provided to an Artificial Intelligence (AI) model to direct its behavior and produce the desired results. +The quality and specificity of the prompt are crucial in obtaining precise and relevant outputs. +A well-designed prompt can ensure that the AI model generates the desired information or completes the intended task effectively. +Some typical prompts include summarization, question answering, text classification, and code generation. While there's techniques and patterns used when building an Azure OpenAI solution and writing prompts, the following is a couple simple prompt examples: @@ -43,15 +54,26 @@ While there's techniques and patterns used when building an Azure OpenAI solutio #### Guidelines for creating robust prompts -While it can be quick to write basic prompts, it can also be difficult to write more complex prompts to ge the AI to generate the responses necessary. When writing prompts, there are three basic guidelines to follow for creating useful prompts: +While it can be quick to write basic prompts, it can also be difficult to write more complex prompts to ge the AI to generate the responses necessary. +When writing prompts, there are three basic guidelines to follow for creating useful prompts: -- **Show and tell** - Make it clear what response is desired either through instructions, examples, or a combination of the two. When ranking a list of items in alphabetical order or to classifying a paragraph by sentiment, include these details in the prompt to provided to the model. -- **Provide quality data** - When building a classifier or get the model to follow a pattern, make sure there are enough examples. Be sure to proofread the examples. The model is smart enough to resolve basic spelling mistakes and still provide a meaningful response. Conversely, the model might assume the mistakes are intentional, which can affect the response. -- **Check the settings** - Probability settings, such as `Temperature` and `Top P`, control how deterministic the model is in generating a response. When asking for a response where there's only one right answer, a lower value should be specified for these settings. When looking for a response that are not obvious and needs additional creativity, use higher values. The most common mistake users make with these settings is assuming they control "cleverness" or "creativity" in the model response. +- **Show and tell** - Make it clear what response is desired either through instructions, examples, or a combination of the two. +When ranking a list of items in alphabetical order or to classifying a paragraph by sentiment, include these details in the prompt to provided to the model. +- **Provide quality data** - When building a classifier or get the model to follow a pattern, make sure there are enough examples. +Be sure to proofread the examples. +The model is smart enough to resolve basic spelling mistakes and still provide a meaningful response. +Conversely, the model might assume the mistakes are intentional, which can affect the response. +- **Check the settings** - Probability settings, such as `Temperature` and `Top P`, control how deterministic the model is in generating a response. +When asking for a response where there's only one right answer, a lower value should be specified for these settings. +When looking for a response that are not obvious and needs additional creativity, use higher values. +The most common mistake users make with these settings is assuming they control "cleverness" or "creativity" in the model response. ### What is prompt engineering -[Prompt engineering](https://learn.microsoft.com/azure/ai-services/openai/concepts/prompt-engineering) is the iterative process of designing, evaluating, and optimizing prompts to produce consistently accurate responses from language models for a particular problem domain. It involves designing and refining the prompts given to an AI model to achieve the desired outputs. Prompt engineers experiment with various prompts, test their effectiveness, and refine them to improve performance. Performance is measured using predefined metrics such as accuracy, relevance, and user satisfaction to assess the impact of prompt engineering. +[Prompt engineering](https://learn.microsoft.com/azure/ai-foundry/openai/concepts/prompt-engineering) is the iterative process of designing, evaluating, and optimizing prompts to produce consistently accurate responses from language models for a particular problem domain. +It involves designing and refining the prompts given to an AI model to achieve the desired outputs. +Prompt engineers experiment with various prompts, test their effectiveness, and refine them to improve performance. +Performance is measured using predefined metrics such as accuracy, relevance, and user satisfaction to assess the impact of prompt engineering. ### General anatomy of a prompt @@ -67,54 +89,87 @@ There are a several components that are used in the anatomy of constructing AI p #### Input -The Input in prompt engineering is the text or query that is provided to the LLM model. The model will use this prompt as the basis for the completion, or response, that is generated. +The Input in prompt engineering is the text or query that is provided to the LLM model. +The model will use this prompt as the basis for the completion, or response, that is generated. ##### Instructions -Instructions are likely the most commonly used prompt component. Instructions are simply that: instructions to the model on what to do. While simple in concept, they can be complex in practice. +Instructions are likely the most commonly used prompt component. +Instructions are simply that: instructions to the model on what to do. +While simple in concept, they can be complex in practice. The following are examples of both simple and more complex instructions: | Instruction Type | Prompt | Completion | | :--- | :--- | :--- | -| Simple | Write an introduction for a weekly newsletter. | Welcome to our weekly newsletter. This week we have a special focus on productivity. We'll be sharing tips and tricks on how to be productive in all areas of your life. We hope you enjoy and please share with your friends! | -| Complex | Write an introduction for a weekly newsletter to my company named Contoso. It should mention the great all hands meeting we had last week and thank the team for all their hard work over the tough past few months. | Welcome to Contoso's weekly newsletter! We had a great all hands meeting last week and I want to thank the team for all their hard work over the tough past few months. We're looking forward to a bright future and I'm confident that with our team's dedication and commitment, we will achieve great things. Thank you for being a part of our team and I look forward to working with you all in the coming weeks and months. Sincerely, [Your name]. | -| Complex | Write an introduction for a weekly newsletter, including the following:
- My company is named contoso
- An exuberant greeting
- Thanking the team for their hard work over a couple tough months
- Positive outlook for the coming quarter
- Signed by the SLT | Hello Contoso team! I wanted to take a moment to thank you for all your hard work over the past few months. It's been a tough couple of months, but you've all pulled together and done an amazing job. I'm really proud of you all. Looking ahead to the next quarter, I'm feeling really positive. I think we're going to achieve some great things. Thanks again, The Senior Leadership Team. | +| Simple | Write an introduction for a weekly newsletter. | Welcome to our weekly newsletter. +This week we have a special focus on productivity. +We'll be sharing tips and tricks on how to be productive in all areas of your life. +We hope you enjoy and please share with your friends! | +| Complex | Write an introduction for a weekly newsletter to my company named Contoso. +It should mention the great all hands meeting we had last week and thank the team for all their hard work over the tough past few months. | Welcome to Contoso's weekly newsletter! +We had a great all hands meeting last week and I want to thank the team for all their hard work over the tough past few months. +We're looking forward to a bright future and I'm confident that with our team's dedication and commitment, we will achieve great things. +Thank you for being a part of our team and I look forward to working with you all in the coming weeks and months. +Sincerely, [Your name]. | +| Complex | Write an introduction for a weekly newsletter, including the following:
- My company is named contoso
- An exuberant greeting
- Thanking the team for their hard work over a couple tough months
- Positive outlook for the coming quarter
- Signed by the SLT | Hello Contoso team! +I wanted to take a moment to thank you for all your hard work over the past few months. +It's been a tough couple of months, but you've all pulled together and done an amazing job. +I'm really proud of you all. +Looking ahead to the next quarter, I'm feeling really positive. +I think we're going to achieve some great things. +Thanks again, The Senior Leadership Team. | ##### Primary Content -Primary content refers to some sort of text that is being processed or transformed by the model. Primary content is typically used in conjunction with instructions. A simple example would be language translation. +Primary content refers to some sort of text that is being processed or transformed by the model. +Primary content is typically used in conjunction with instructions. +A simple example would be language translation. | Prompt | Completion | | :--- | :--- | | Can you please tell me how to get to the museum?
Translate to French: | `Pouvez-vous s'il vous plaît me dire comment aller au musée?` | -Primary content can also be much longer. For example, the primary content could pass the introduction section of text content that could be hundreds of words long. Additionally, the primary content could be int he form of structured data as well, such as in JSON or TSV format. +Primary content can also be much longer. +For example, the primary content could pass the introduction section of text content that could be hundreds of words long. +Additionally, the primary content could be int he form of structured data as well, such as in JSON or TSV format. ##### Supporting Content -Supporting content is information that the model can utilize to influence the output in some way. It differs from primary content in that it is not the main target of task, but it is typically used along with primary content. Common examples include contextual information such as the current date, the name of the user, user preferences, and so on. +Supporting content is information that the model can utilize to influence the output in some way. +It differs from primary content in that it is not the main target of task, but it is typically used along with primary content. +Common examples include contextual information such as the current date, the name of the user, user preferences, and so on. The following are example prompts and possible completions to show what a prompt with and without supporting content might look like: | Supporting Content | Prompt | Completion | | :--- | :--- | :--- | | Without | Workshop list: -Title Description
Planning Workshop Kickoff Please join us to kick off our workshop! Hear from our leaders about cross group priorities and learnings for the year ahead. The invitation to this event has already been sent to everyone in […]
Summarize the workshops above, grouped by topic: | Workshop list:
Planning Workshop Kickoff
- Planning Workshop Wrap-up
- Core MeetUp
- Customer
Driven Perspectives
- Impact and Key Focus Areas
- Best practices
- Accelerating app adoption […] | -| With | Workshop list:
Title Description
Planning Workshop Kickoff Please join us to kick off our workshop! Hear from our leaders about cross group priorities and learnings for the year ahead. The invitation to this event has already been sent to everyone in […]
My Important Topics: prompt engineering, search, GPT models
Summarize the workshops above, grouped by my important topics: | 1. Prompt Engineering:
- New prompt engineering tips
- The art of prompt engineering
2. Search:
- Intro to vector search with embedding
3. GPT Models:
- Intro to GPT-4
- GPT-35-Turbo in-depth. | - -The above prompt examples are truncated with `[…]` for brevity. Writing similar prompts would contain more text. +Title Description
Planning Workshop Kickoff Please join us to kick off our workshop! +Hear from our leaders about cross group priorities and learnings for the year ahead. +The invitation to this event has already been sent to everyone in […]
Summarize the workshops above, grouped by topic: | Workshop list:
Planning Workshop Kickoff
- Planning Workshop Wrap-up
- Core MeetUp
- Customer
Driven Perspectives
- Impact and Key Focus Areas
- Best practices
- Accelerating app adoption […] | +| With | Workshop list:
Title Description
Planning Workshop Kickoff Please join us to kick off our workshop! +Hear from our leaders about cross group priorities and learnings for the year ahead. +The invitation to this event has already been sent to everyone in […]
My Important Topics: prompt engineering, search, GPT models
Summarize the workshops above, grouped by my important topics: | 1. +Prompt Engineering:
- New prompt engineering tips
- The art of prompt engineering
2. +Search:
- Intro to vector search with embedding
3. +GPT Models:
- Intro to GPT-4
- GPT-35-Turbo in-depth. | + +The above prompt examples are truncated with `[…]` for brevity. +Writing similar prompts would contain more text. ##### System message -System message, also called a system prompt, is used by developers of a system to restrict the LLM to a specific set of constraints. This is useful when building enterprise solutions that integrate Azure OpenAI so the AI completion will be restricted to the focus of the enterprise data it's integrated with. +System message, also called a system prompt, is used by developers of a system to restrict the LLM to a specific set of constraints. +This is useful when building enterprise solutions that integrate Azure OpenAI so the AI completion will be restricted to the focus of the enterprise data it's integrated with. The following is an example system message that could be used to constrain the LLM in an enterprise solution: ```text You are a helpful, fun and friendly sales assistant for Cosmic Works, a bicycle and bicycle accessories store. -Your name is Cosmo. +Do not include citations or citation numbers in your responses. +Do not include emojis. You are designed to answer questions about the products that Cosmic Works sells, the customers that buy them, and the sales orders that are placed by customers. @@ -127,49 +182,69 @@ If a question is not related to Cosmic Works products, customers, or sales order #### Output -The Output is the completion, or response, from the LLM returned as a result to the input prompt given. When an input prompt is given, the language model will process the information and generate an output in the form of text. The text response is the output. +The Output is the completion, or response, from the LLM returned as a result to the input prompt given. +When an input prompt is given, the language model will process the information and generate an output in the form of text. +The text response is the output. ## Standard Patterns ### Retrieval Augmentation Generation (RAG) -[Retrieval Augmentation Generation (RAG)](https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview) is an architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system that provides grounding data. Adding an information retrieval system provides control over grounding data used by an LLM when it formulates a response. +[Retrieval Augmentation Generation (RAG)](https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview) is an architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system that provides grounding data. +Adding an information retrieval system provides control over grounding data used by an LLM when it formulates a response. -GPT language models can be fine-tuned to achieve several common tasks such as sentiment analysis and named entity recognition. These tasks generally don't require additional background knowledge. +GPT language models can be fine-tuned to achieve several common tasks such as sentiment analysis and named entity recognition. +These tasks generally don't require additional background knowledge. -The RAG pattern facilitates bringing private proprietary knowledge to the model so that it can perform Question Answering over this content. Remember that Large Language Models are indexed only on public information. For an enterprise solution, RAG architecture means that the generative AI is constrained to enterprise content sourced from vectorized documents, images, audio, and video. +The RAG pattern facilitates bringing private proprietary knowledge to the model so that it can perform Question Answering over this content. +Remember that Large Language Models are indexed only on public information. +For an enterprise solution, RAG architecture means that the generative AI is constrained to enterprise content sourced from vectorized documents, images, audio, and video. Because the RAG technique accesses external knowledge sources to complete tasks, it enables more factual consistency, improves the reliability of the generated responses, and helps to mitigate the problem of "_hallucination_". -In some cases, the RAG process involves a technique called vectorization on the proprietary data. The user prompt is compared to the vector store and only the most relevant/matching pieces of information are returned and stuffed into prompt for the LLM to reason over and provide an answer. The next set of demos will go into this further. +In some cases, the RAG process involves a technique called vectorization on the proprietary data. +The user prompt is compared to the vector store and only the most relevant/matching pieces of information are returned and stuffed into prompt for the LLM to reason over and provide an answer. +The next set of demos will go into this further. ### Chain of Thought (CoT) -Instead of splitting a task into smaller steps, with [Chain of Thought (CoT)](https://learn.microsoft.com/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions#chain-of-thought-prompting) the model response is instructed to proceed step-by-step and present all the steps involved. Doing so reduces the possibility of inaccuracy of outcomes and makes assessing the model response easier. +Instead of splitting a task into smaller steps, with [Chain of Thought (CoT)](https://learn.microsoft.com/azure/ai-foundry/openai/concepts/prompt-engineering?view=foundry-classic#chain-of-thought-prompting) the model response is instructed to proceed step-by-step and present all the steps involved. +Doing so reduces the possibility of inaccuracy of outcomes and makes assessing the model response easier. | System message | Prompt | Response | | :--- | :--- | :--- | -| You are an AI assistant that helps people find information. | Who was the most decorated (maximum medals) individual athlete in the Olympic games that were held at Sydney? Take a step-by-step approach in your response, cite sources and give reasoning before sharing final answer in the below format: ANSWER is: `` | Step 1: Determine the year of the Sydney Olympic games. The Sydney Olympic games were held in the year 2000.

Step 2: Find a list of individual athletes who won multiple medals in the 2000 Sydney Olympics.

According to the official website of the Olympic Games, there were several individual athletes who won multiple medals in the 2000 Sydney Olympics. Some of the notable names include:
- Ian Thorpe (Australia) - 3 gold, 2 silver, 0 bronze
- Pieter van den Hoogenband (Netherlands) - 2 gold, 0 silver, 2 bronze
- Lenny Krayzelburg (USA) - 3 gold, 0 silver, 0 bronze
- Inge de Bruijn (Netherlands) - 3 gold, 1 silver, 0 bronze

Step 3: Determine the athlete with the most medals. After reviewing the list of individual athletes who won multiple medals in the 2000 Sydney Olympics, we can see that the athlete with the most medals is Ian Thorpe from Australia, who won a total of 5 medals (3 gold and 2 silver).

Therefore, the ANSWER is: Ian Thorpe. | +| You are an AI assistant that helps people find information. | Who was the most decorated (maximum medals) individual athlete in the Olympic games that were held at Sydney? +Take a step-by-step approach in your response, cite sources and give reasoning before sharing final answer in the below format: ANSWER is: `` | Step 1: Determine the year of the Sydney Olympic games. +The Sydney Olympic games were held in the year 2000.

Step 2: Find a list of individual athletes who won multiple medals in the 2000 Sydney Olympics.

According to the official website of the Olympic Games, there were several individual athletes who won multiple medals in the 2000 Sydney Olympics. +Some of the notable names include:
- Ian Thorpe (Australia) - 3 gold, 2 silver, 0 bronze
- Pieter van den Hoogenband (Netherlands) - 2 gold, 0 silver, 2 bronze
- Lenny Krayzelburg (USA) - 3 gold, 0 silver, 0 bronze
- Inge de Bruijn (Netherlands) - 3 gold, 1 silver, 0 bronze

Step 3: Determine the athlete with the most medals. +After reviewing the list of individual athletes who won multiple medals in the 2000 Sydney Olympics, we can see that the athlete with the most medals is Ian Thorpe from Australia, who won a total of 5 medals (3 gold and 2 silver).

Therefore, the ANSWER is: Ian Thorpe. | ### ReAct -Reasoning and Acting (ReAct) leverages LLMs to generate both reasoning traces and task-specific actions. ReAct combines these to allow for greater synergy between the two. Reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external data sources to gather additional information. When using LLMs to answer questions and verify facts, ReAct is used to overcome issues of hallucination and error propagation by interacting with the additional usage of external data sources. +Reasoning and Acting (ReAct) leverages LLMs to generate both reasoning traces and task-specific actions. +ReAct combines these to allow for greater synergy between the two. +Reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external data sources to gather additional information. +When using LLMs to answer questions and verify facts, ReAct is used to overcome issues of hallucination and error propagation by interacting with the additional usage of external data sources. ### Others #### Zero-shot prompting -Zero-shot prompting is what we would consider the “default”. This is when we provide no examples of inputs/expected outputs to the model to work with. We’re leaving it up to the model to decipher what is needed and how to output it from the instructions. +Zero-shot prompting is what we would consider the “default”. +This is when we provide no examples of inputs/expected outputs to the model to work with. +We’re leaving it up to the model to decipher what is needed and how to output it from the instructions. Example: | Prompt | Completion | | :--- | :--- | -| Headline: Coach confident injury won't derail Warriors
Topic: | The coach is confident that the injury won't derail the Warriors' season. The team is still focused on their goals and that they will continue to work hard to achieve them. | +| Headline: Coach confident injury won't derail Warriors
Topic: | The coach is confident that the injury won't derail the Warriors' season. +The team is still focused on their goals and that they will continue to work hard to achieve them. | #### Few-shot prompting -Few-shot prompting is a technique that enables in-context learning for the LLM by providing examples or further information within the prompt to steer the model to generate a better response. Providing additional information in the prompt helps particularly in areas where the data used to train the model isn't enough to generate the desired output from the model. +Few-shot prompting is a technique that enables in-context learning for the LLM by providing examples or further information within the prompt to steer the model to generate a better response. +Providing additional information in the prompt helps particularly in areas where the data used to train the model isn't enough to generate the desired output from the model. Example: diff --git a/05_Explore_OpenAI_models/README.md b/05_Explore_OpenAI_models/README.md index 8b49339..cd4c882 100644 --- a/05_Explore_OpenAI_models/README.md +++ b/05_Explore_OpenAI_models/README.md @@ -2,25 +2,44 @@ ## Azure OpenAI Models -Azure OpenAI is powered by a diverse set of models with different capabilities. - -| Model | Description | -| -- | --- | -| GPT-4 | A set of models that improve on GPT-3.5 and can understand and generate natural language and code. | -| GPT-3.5 | A set of models that improve on GPT-3 and can understand and generate natural language and code. | +Azure Open AI is powered by a diverse set of models with different capabilities [Azure OpenAI models]https://learn.microsoft.com/azure/ai-foundry/foundry-models/concepts/models-sold-directly-by-azure). + +| Model | Description | +| ---------- | ----------- | +| GPT-5.1 series (gpt-5.1, gpt-5.1-chat, gpt-5.1-codex, gpt-5.1-codex-mini) | The latest general purpose models from Open AI | +| Video generation (sora-2, sora) | A model that can generate original video scenes from text instructions. | +| GPT-5 series (gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-chat) | General purpose models that improve upon GPT 4.1 models | +| gpt-oss | open-weight reasoning models | +| codex-mini | A Fine-tuned version of o4-mini. | +| GPT-4.1 series (gpt-4.1, gpt-4.1-mini, gpt-4.1-nano) | General purpose models that improve upon GPT 4 models | +| model-router | A model that intelligently selects from a set of underlying chat models to respond to a given prompt. | +| computer-use-preview | An experimental model trained for use with the Responses API computer use tool. | +| o-series models | Reasoning models with advanced problem solving and increased focus and capability. | +| GPT-4o, GPT-4o mini, and GPT-4 Turbo | Capable Azure OpenAI models with multimodal versions, which can accept both text and images as input. | | Embeddings | A set of models that can convert text into numerical vector form to facilitate text similarity. | -| DALL-E | A series of models that can generate original images from natural language. | -| Whisper | A series of models that can transcribe and translate speech to text. | +| Image generation (dall-e-2, gpt-image-1) | A series of models that can generate original images from natural language. | +| Audio (gpt-realtime, gpt-audio, etc) | A series of models for speech to text, translation, and text to speech. | +| Whisper | A series of models that can transcribe and translate speech to text. | -### GPT-4 and GPT-3.5 Models +NOTE that model availability varies by region. +You can use [Azure OpenAI models]https://learn.microsoft.com/azure/ai-foundry/foundry-models/concepts/models-sold-directly-by-azure) to determine the regional availability of any model. -GPT-4 can solve difficult problems with greater accuracy than any of OpenAI's previous models. Like GPT-3.5 Turbo, GPT-4 is optimized for chat and works well for traditional completions tasks. +### GPT models -The GPT-35-Turbo and GPT-4 models are language models that are optimized for conversational interfaces. The models behave differently than the older GPT-3 models. Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. However, the GPT-35-Turbo and GPT-4 models are conversation-in and message-out. The models expect input formatted in a specific chat-like transcript format, and return a completion that represents a model-written message in the chat. While this format was designed specifically for multi-turn conversations, it can also work well for non-chat scenarios too. +All of the newer GPT models are language models that are optimized for conversational interfaces. +Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. +However, GPT-35-Turbo and GPT-4 models and beyond are conversation-in and message-out. +The models expect input formatted in a specific chat-like transcript format, and return a completion that represents a model-written message in the chat. +While this format was designed specifically for multi-turn conversations, it can also work well for non-chat scenarios. ### Embeddings -Embeddings, such as the `text-embedding-ada-002` model, measure the relatedness of text strings. +Embedding models are designed to turn language into a numerical representation called a vector. +You can measure how similar chunks of language are by measuring their cosine similarity (distance). +`text-embedding-3-large` is the latest and most capable embedding model. +NOTE: You can't upgrade between embeddings models. To move from using text-embedding-ada-002 to text-embedding-3-large, you need to generate new embeddings. + +OpenAI reports that testing shows that both the large and small third generation embeddings models offer better average multi-language retrieval performance with the MIRACL benchmark. They still maintain performance for English tasks with the MTEB benchmark. Embeddings are commonly used for the following: @@ -37,24 +56,34 @@ DALL-E is a model that can generate an original images from a natural language t ### Whisper -Whisper is a speech recognition model, designed for general-purpose applications. Trained on an extensive dataset encompassing diverse audio inputs, and operates as a multi-tasking model capable of executing tasks like multilingual speech recognition, speech translation, and language identification. +Whisper is a speech recognition model, designed for general-purpose applications. +Trained on an extensive dataset encompassing diverse audio inputs, and operates as a multi-tasking model capable of executing tasks like multilingual speech recognition, speech translation, and language identification. ## Selecting an LLM -Before a Large Language Model (LLM) can be implemented into a solution, an LLM model must be chosen. For this the business use case and other aspects to the overall goal of the AI solution will need to be defined. +Before a Large Language Model (LLM) can be implemented into a solution, an LLM model must be chosen. +For this the business use case and other aspects to the overall goal of the AI solution will need to be defined. Once the business goals of the solution are known, there are a few key considerations to think about: -- **Business Use Case** - What are the specific tasks the business needs the AI solution to perform? Each LLM is designed for different goals, such as text generation, language translation, image generation, answering questions, code generation, etc. -- **Pricing** - For cases where there may be multiple LLMs to choose from, the [pricing](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) of the LLM could be a factor to consider. For example, when choosing between GPT-3.5 or GPT-4, it may be worth to consider that the overall cost of GPT-4 may be higher than GPT-3.5 for the solution since GPT-4 requires more compute power behind the scenes than GPT-3.5 -- **Accuracy** - For cases where there may be multiple LLMs to choose from, the comparison of accuracy between them may be a factor to consider. For example, GPT-4 offers improvements over GPT-3.5 and depending on the use case, GPT-4 may provide increased accuracy. -- **Quotas and limits** - The Azure OpenAI service does have [quotas and limits](https://learn.microsoft.com/azure/ai-services/openai/quotas-limits) on using the service. This may affect the performance and pricing of the AI solution. Additionally, some of quotas and limits may vary depending on the Azure Region that is used to host the Azure OpenAI service. The potential impact of these on the pricing and performance of the solution will want to be considered in the design phase of the solution. +- **Business Use Case** - What are the specific tasks the business needs the AI solution to perform? +Each LLM is designed for different goals, such as text generation, language translation, image generation, answering questions, code generation, etc. +- **Pricing** - For cases where there may be multiple LLMs to choose from, the [pricing of Azure Open AI](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) could be a factor to consider. +For example, when choosing between older models and newer ones, it may be worth to consider that the overall cost of newer models may be higher than an older one for the solution since newer ones require more compute power behind the scenes than older models. +Also, you should consider using a mini model when performing tasks which require less reaasoning. +- **Accuracy** - For cases where there may be multiple LLMs to choose from, the comparison of accuracy between them may be a factor to consider. +For example, newer models may offer improvements over older ones and depending on the use case, new models may provide increased accuracy. +- **Quotas and limits** - There are [Azure Open AI quotas and limits](https://learn.microsoft.com/azure/ai-foundry/openai/quotas-limits) on using the service. This may affect the performance and pricing of the AI solution. +Additionally, some of quotas and limits may vary depending on the Azure Region that is used to host the Azure OpenAI service. +The potential impact of these on the pricing and performance of the solution will want to be considered in the design phase of the solution. ## Do I use an out-of-the-box model or a fine-tuned model? -A base model is a model that hasn't been customized or fine-tuned for a specific use case. Fine-tuned models are customized versions of base models where a model's weights are trained on a unique set of prompts. Fine-tuned models achieve better results on a wider number of tasks without needing to provide detailed examples for in-context learning as part of the completion prompt. +A base model is a model that hasn't been customized or fine-tuned for a specific use case. +Fine-tuned models are customized versions of base models where a model's weights are trained on a unique set of prompts. +Fine-tuned models achieve better results on a wider number of tasks without needing to provide detailed examples for in-context learning as part of the completion prompt. -The [fine-tuning guide](https://learn.microsoft.com/azure/ai-services/openai/how-to/fine-tuning) can be referenced for more information. +The [fine-tuning guide](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/fine-tuning) can be referenced for more information. ## Explore and use Azure OpenAI models from code @@ -64,6 +93,6 @@ The `key` and `endpoint` necessary to make API calls to Azure OpenAI can be loca ## Lab: Explore and use Azure OpenAI models from code -This labs demonstrates using an Azure OpenAI model to obtain a completion response using the Node.js runtime with the JavaScript programming language. +This lab demonstrates using an Azure OpenAI model to obtain a completion response using Node.js runtime and the JavaScript. -Visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/explore_and_use_models/README.md). +Visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/lab_0_explore_and_use_models/README.md). diff --git a/06_Provision_Azure_Resources/README.md b/06_Provision_Azure_Resources/README.md index f00abe8..b864a49 100644 --- a/06_Provision_Azure_Resources/README.md +++ b/06_Provision_Azure_Resources/README.md @@ -1,27 +1,32 @@ # Provision Azure resources (Azure Cosmos DB workspace, Azure OpenAI, etc.) -As the guide walks you through the concepts of integrating vCore-based Azure Cosmos DB for MongoDB and Azure OpenAI, the hands-on labs will also guide you through building a sample solution. The focus of this guide and the labs is limited to vCore-based Azure Cosmos DB for MongoDB, Vector Search, Azure OpenAI, and the Python programming language. With this focus, the labs include an Azure Bicep template that will deploy the following Azure resources the solution will be deployed to: +As the guide walks you through the concepts of integrating Azure DocumentDB and Azure OpenAI, the hands-on labs will also guide you through building a sample solution. +The focus of this guide and the labs is limited to Azure DocumentDB, vector search capabilities powered by DiskANN, Azure OpenAI, and the Node.js programming language. +With this focus, the labs include an Azure Bicep template that will deploy the following Azure resources the solution will be deployed to: - Azure Resource Group -- vCore-based Azure Cosmos DB for MongoDB +- Azure DocumentDB - Azure OpenAI - ChatGPT-3.5 `completions` model - - text-embedding-ada-002 model `embeddings` model + - text-embedding-3-small model `embeddings` model - Azure App Service - for hosting the front-end, static SPA web application written in React -- Azure Container App - for hosting the back-end, Node.js API application written in JavaScript +- Azure Container App - for hosting the Node.js Back-end API application written in JavaScript - Azure Container Registry - to host Docker images of backend, API application ## Architecture Diagram ![Solution architecture diagram showing how the Azure services deployed are connected](media/architecture.jpg) -Once the Azure resources are provisioned, this guide will walk you through everything that is necessary to build the Node.js Back-end API application. +Once the Azure resources are provisioned, this guide will walk you through everything that is necessary to build the Node.js Back-end API application written in JavaScript. -The Front-end Web App is a static SPA application written in React. Since React is outside the scope of this guide, the Front-end Web App is pre-built for you and will be configured automatically on deployment. You do not need any experience with React in order to complete the labs in this guide. +The Front-end Web App is a static SPA application written in React. +Since React is outside the scope of this guide, the Front-end Web App is pre-built for you and will be configured automatically on deployment. +You do not need any experience with React in order to complete the labs in this guide. ## Lab - Provision Azure Resources -This lab will walk you through deploying the Azure resources necessary for the solution built in this guide. The deployment will be done using an Azure Bicep template that is configured to provision all the necessary resources. +This lab will walk you through deploying the Azure resources necessary for the solution built in this guide. +The deployment will be done using an Azure Bicep template that is configured to provision all the necessary resources. > **Note**: You will need an Azure Subscription and have the necessary permissions to provision the Azure resources. diff --git a/07_Create_First_Cosmos_DB_Project/README.md b/07_Create_First_Cosmos_DB_Project/README.md index 9ce8932..79756a9 100644 --- a/07_Create_First_Cosmos_DB_Project/README.md +++ b/07_Create_First_Cosmos_DB_Project/README.md @@ -1,48 +1,58 @@ -# Create your first Cosmos DB project +# Create your first Azure Cosmos DB project -This section will cover how to create your first Cosmos DB project. We'll create a simple application to demonstrate the basic CRUD operations. We'll also cover using the Azure Cosmos DB Emulator to test code locally. +This section will cover how to create your first Azure Cosmos DB project. +We'll create a simple application to demonstrate the basic CRUD operations. +We'll also cover using a local DocumentDB Docker container to test code locally. -## Emulator support +## Local development support -Azure Cosmos DB has an emulator that can be used to develop code locally. The emulator supports the API for NoSQL and the API for MongoDB. The use of the emulator does not require an Azure subscription, nor does it incur any costs, so it is ideal for local development and testing. The Azure Cosmos DB emulator can also be utilized with unit tests in a [GitHub Actions CI workflow](https://learn.microsoft.com/azure/cosmos-db/how-to-develop-emulator?tabs=windows%2Cpython&pivots=api-mongodb#use-the-emulator-in-a-github-actions-ci-workflow). +For local development and testing, the [DocumentDB Docker container](https://github.com/microsoft/documentdb) can be used. +This approach does not require an Azure subscription and is ideal for local development. -There is not 100% feature parity between the emulator and the cloud service. Visit the [Azure Cosmos DB emulator](https://learn.microsoft.com/azure/cosmos-db/emulator) documentation for more details. +Run the DocumentDB container using Docker: -For Windows machines, the emulator can be installed via an installer. There is a Windows container using Docker available. However, it does not currently support the API for Mongo DB. A Docker image is also available for Linux that does support the API for Mongo DB. +```console +docker pull ghcr.io/documentdb/documentdb/documentdb-local:latest -Learn more about the pre-requisites and installation of the emulator [here](https://learn.microsoft.com/azure/cosmos-db/how-to-develop-emulator?tabs=windows%2Cpython&pivots=api-mongodb). - ->**NOTE**: When using the Azure CosmosDB emulator using the API for MongoDB it must be started with the [MongoDB endpoint options enabled](https://learn.microsoft.com/azure/cosmos-db/how-to-develop-emulator?tabs=windows%2Cpython&pivots=api-mongodb#start-the-emulator) at the command-line. +docker run -dt -p 10260:10260 --name documentdb-local ghcr.io/documentdb/documentdb/documentdb-local:latest --username --password +``` -**The Azure Cosmos DB emulator does not support vector search. To complete the vector search and AI-related labs, a vCore-based Azure Cosmos DB for MongoDB account in Azure is required.** +### Retrieving the connection string for local development -## Authentication +The connection string for the local DocumentDB container is: +`mongodb://:@localhost:10260/?tls=true&tlsAllowInvalidCertificates=true` -Authentication to Azure Cosmos DB API for Mongo DB uses a connection string. The connection string is a URL that contains the authentication information for the Azure Cosmos DB account or local emulator. The username and password used when provisioning the Azure Cosmos DB API for MongoDB service are used in the connection string when authenticating to Azure. +Replace `` and `` with the credentials you specified when starting the container. -### Retrieving the connection string from the Cosmos DB Emulator +DocumentDB has vector search capabilities out of the box, allowing you to run your AI workloads locally. +For more information on using vector search locally, see the [DocumentDB documentation](https://documentdb.io/docs). -The splash screen or **Quickstart** section of the Cosmos DB Emulator will display the connection string. Access this screen through the following URL: `https://localhost:8081/_explorer/index.html`. +## Authentication -![The Azure Cosmos DB emulator screen displays with the local host url, the Quickstart tab, and the Mongo connection string highlighted.](media/emulator_connection_string.png) +Authentication to Azure DocumentDB uses a connection string. +The connection string is a URL that contains the authentication information for the Azure DocumentDB cluster or local development environment. +The username and password used when provisioning the Azure DocumentDB service are used in the connection string when authenticating to Azure. ### Retrieving the connection string from the Azure portal -Retrieve the connection string from the Azure portal by navigating to the Azure Cosmos DB account and selecting the **Connection String** menu item on the left-hand side of the screen. The connection string contains tokens for the username and password that must be replaced with the username and password used when provisioning the Azure Cosmos DB API for MongoDB service. +Retrieve the connection string from the Azure portal by navigating to the Azure DocumentDB cluster and selecting the **Connection String** menu item on the left-hand side of the screen. +The connection string contains tokens for the username and password that must be replaced with the username and password used when provisioning the Azure DocumentDB service. -![The Azure CosmosDb API for MongoDB Connection strings screen displays with the copy button next to the connection string highlighted.](media/azure_connection_string.png) +![The Azure DocumentDB Connection strings screen displays with the copy button next to the connection string highlighted.](media/azure_connection_string.png) -## Lab - Create your first Cosmos DB for MongoDB application +## Lab - Create your first Azure DocumentDB application -In this lab, we'll create a Cosmos DB for the MongoDB application using the **mongodb** NPM package that includes the MongoDB Node.js Driver and its dependencies. Both the Azure Cosmos DB Emulator and Azure Cosmos DB account in Azure are supported for completion of this lab. +In this lab, we'll create an Azure DocumentDB application using the **mongodb** NPM package that includes the MongoDB Node.js Driver and its dependencies. +Both the Azure Cosmos DB Emulator and Azure DocumentDB cluster in Azure are supported for completion of this lab. -Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/first_cosmos_db_application/README.md). +Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/lab_1_first_application/README.md). The following concepts are covered in detail in this lab: -### Creating a MongoDB database client +### Creating a database client -The `mongodb` NPM package is used to create a MongoDB database client. The client enables both DDL (data definition language) and DML (data manipulation language) operations. +The `mongodb` package is used to create an Azure DocumentDB database client. +The client enables both DDL (data definition language) and DML (data manipulation language) operations. ```javascript const client = new MongoClient(process.env.MONGODB_URI); @@ -50,48 +60,55 @@ const client = new MongoClient(process.env.MONGODB_URI); ### Creating a database -When using the `mongodb` client, the creation of a database is automatic when referenced. No specific api calls to create a database are required, if a database already exists, a reference to the database is returned. +When using the MongoClient, the creation of a database is automatic when referenced. +No specific api calls to create a database are required, if a database already exists, a reference to the database is returned. ->**Note:**: That the creation of databases and collections are lazy, meaning they will not be created until a document is inserted into a collection. +> **Note:**: That the creation of databases and collections are lazy, meaning they will not be created until a document is inserted into a collection. ```javascript -const db = client.db('cosmic_works'); +const db = client.db("cosmic_works"); ``` ### Creating a collection -Similar behavior to the creation of a database is experienced when creating a collection. If the collection does not exist, it will be created once a document is inserted into the collection. +Similar behavior to the creation of a database is experienced when creating a collection. +If the collection does not exist, it will be created once a document is inserted into the collection. ```javascript -const products = db.collection('products'); +const products = db.collection("products"); ``` ### Creating a document -The `insertOne` method is used to insert a document into a collection. The document is a product document. +The `insertOne` method is used to insert a document into a collection. +The document is a product dictionary object. ```javascript const result = await products.insertOne(product); +const productId = result.insertedId; ``` ### Reading a document -The `findOne` method is used to retrieve a single document from a collection. The method returns the product document. +The `findOne` method is used to retrieve a single document from a collection. +The method returns a product dictionary object. ```javascript -const product = await products.findOne({ _id: '2BA4A26C-A8DB-4645-BEB9-F7D42F50262E' }); +const product = await products.findOne({ _id: productId }); ``` ### Updating a document -The `findOneAndUpdate` method is used to update a single document in a collection. The method returns the updated document object. +The `findOneAndUpdate` method is used to update a single document in a collection. +The method returns a product dictionary object. ```javascript -const options = { returnDocument: 'after' }; +const options = { returnDocument: "after" }; const updated = await products.findOneAndUpdate( - { _id: '2BA4A26C-A8DB-4645-BEB9-F7D42F50262E' }, - { $set: { price: 14242.42 } }, - options); + { _id: productId }, + { $set: { price: 14242.42 } }, + options +); ``` ### Deleting a document @@ -99,13 +116,15 @@ const updated = await products.findOneAndUpdate( The `deleteOne` method is used to delete a single document from a collection. ```javascript -const result = await products.deleteOne({ _id: '2BA4A26C-A8DB-4645-BEB9-F7D42F50262E' }); +const result = await products.deleteOne({ _id: productId }); ``` ### Querying documents -The `find` method is used to query documents from a collection. The method returns a cursor object that can be converted to an array using the `toArray` method. +The `find` method is used to query documents from a collection. +The method returns a cursor object that can be converted to an array using the `toArray` method. ```javascript const allProducts = await products.find({}).toArray(); ``` + diff --git a/08_Load_Data/README.md b/08_Load_Data/README.md index 6d26b00..151e2a0 100644 --- a/08_Load_Data/README.md +++ b/08_Load_Data/README.md @@ -1,36 +1,51 @@ -# Load data into Azure Cosmos DB API for MongoDB +# Load data into Azure DocumentDB -The previous lab demonstrated how to add data to a collection individually. This lab will demonstrate how to load data using bulk operations into multiple collections. This data will be used in subsequent labs to explain further the capabilities of Azure Cosmos DB API for MongoDB with respect to AI. +The previous lab demonstrated how to add data to a collection individually. +This lab will demonstrate how to load data using bulk operations into multiple collections. +This data will be used in subsequent labs to explain further the capabilities of Azure DocumentDB with respect to AI. -When loading data, bulk operations are preferred over adding each document individually. Bulk operations involve performing multiple database operations as a batch rather than executing them simultaneously. This approach is more efficient and provides several benefits: +When loading data, bulk operations are preferred over adding each document individually. +Bulk operations involve performing multiple database operations as a batch rather than executing them serially. +This approach is more efficient and provides several benefits: -1. Performance: By issuing load operations in bulk, the lab can significantly reduce the overhead of network round-trips and database operations. This results in faster data loading and improved overall performance. +1. Performance: By issuing load operations in bulk, the lab can significantly reduce the overhead of network round-trips and database operations. + This results in faster data loading and improved overall performance. -2. Scalability: Bulk operations allow the lab to handle large volumes of data efficiently. They can quickly process and load a substantial amount of customer, product, and sales data, enabling them to scale their operations as needed. +2. Scalability: Bulk operations allow the lab to handle large volumes of data efficiently. + They can quickly process and load a substantial amount of customer, product, and sales data, enabling them to scale their operations as needed. -3. Atomicity: Bulk operations ensure that all database changes within a batch are treated as a single transaction. The entire batch can be rolled back if any document fails to load, maintaining data integrity and consistency. +3. Atomicity: Bulk operations ensure that all database changes within a batch are treated as a single transaction. + The entire batch can be rolled back if any document fails to load, maintaining data integrity and consistency. -4. Simplified code logic: By using bulk operations, the lab can simplify its code logic and reduce the number of database queries. This results in cleaner, more manageable code and reduces the likelihood of errors or inconsistencies. +4. Simplified code logic: By using bulk operations, the lab can simplify its code logic and reduce the number of database queries. + This results in cleaner, more manageable code and reduces the likelihood of errors or inconsistencies. -## Lab - Load data into Azure Cosmos DB API for MongoDB collections +## Lab - Load data into Azure DocumentDB collections -This lab will load the Cosmic Works Customer, Product, and Sales data into Azure Cosmos DB API for MongoDB collections using bulk operations. Both the Azure Cosmos DB Emulator and Azure Cosmos DB account in Azure are supported for completion of this lab. +This lab will load the Cosmic Works Customer, Product, and Sales data into Azure Cosmos DB API for MongoDB collections using bulk operations. +Both the Azure Cosmos DB Emulator and Azure Cosmos DB account in Azure are supported for completion of this lab. Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/08_Load_Data/README.md). -This lab demonstrates the use of bulk operations to load product, customer, and sales data into Azure Cosmos DB API for MongoDB collections. As an example, the following code snippet inserts product data using the `bulkWrite` method allowing for insert functionality using the `InsertOne` operation. The bulkWrite method is used to perform multiple write operations in a single batch, write operations can include a mixture of insert, update, and delete operations: +This lab demonstrates the use of bulk operations to load product, customer, and sales data into Azure Cosmos DB API for MongoDB collections. +The bulkWrite method is used to perform multiple write operations in a single batch, write operations can include a mixture of insert, update, and delete operations. +As an example, the following code snippet inserts product data using the `bulkWrite` method allowing for upsert functionality using the `updateOne` method: ```javascript var result = await productCollection.bulkWrite( - productData.map((product) => ({ - insertOne: { - document: product - } - })) - ); + productData.map((product) => ({ + updateOne: { + filter: { sku: product.sku }, + update: { $set: product }, + upsert: true, + }, + })) +); ``` -The lab continues with bulk loading customer and sales data, this time with the `insertMany` method. The insertMany method is used to insert multiple documents into a collection, it differs from the bulkWrite method in that it only supports insert operations. The following code snippet demonstrates the use of the simplified `insertMany` method to insert customer data: +The lab continues with bulk loading customer and sales data, this time with the `insertMany` method. +The insertMany method is used to insert multiple documents into a collection, it differs from the bulkWrite method in that it only supports insert operations. +The following code snippet demonstrates the use of the simplified `insertMany` method to insert customer data: ```javascript var result = await customerCollection.insertMany(customerData); diff --git a/09_Vector_Search_Cosmos_DB/README.md b/09_Vector_Search_Cosmos_DB/README.md index 2eb9223..532c515 100644 --- a/09_Vector_Search_Cosmos_DB/README.md +++ b/09_Vector_Search_Cosmos_DB/README.md @@ -1,34 +1,57 @@ -# Use vector search on embeddings in vCore-based Azure Cosmos DB for MongoDB +# Use vector search on embeddings in Azure DocumentDB ->**NOTE**: vCore-based Azure Cosmos DB for MongoDB supports vector search on embeddings. This functionality is not supported on RUs-based accounts. +> **NOTE**: Azure DocumentDB supports vector search on embeddings. +This functionality is not supported on RUs-based accounts. ## Embeddings and vector search -Embedding is a way of serializing the semantic meaning of data into a vector representation. Because the generated vector embedding represents the semantic meaning, it means that when it is searched, it can find similar data based on the semantic meaning of the data rather than exact text. Data can come from many sources, including text, images, audio, and video. Because the data is represented as a vector, vector search can, therefore, find similar data across all different types of data. +Embedding is a way of serializing the semantic meaning of data into a vector representation. +Because the generated vector embedding represents the semantic meaning, it means that when it is searched, it can find similar data based on the semantic meaning of the data rather than exact text. +Data can come from many sources, including text, images, audio, and video. +Because the data is represented as a vector, vector search can, therefore, find similar data across all different types of data. -Embeddings are created by sending data to an embedding model, where it is transformed into a vector, which then can be stored as a vector field within its source document in vCore-based Azure Cosmos DB for MongoDB. vCore-based Azure Cosmos DB for MongoDB supports the creation of vector search indexes on top of these vector fields. A vector search index is a collection of vectors in [latent space](https://idl.cs.washington.edu/papers/latent-space-cartography/) that enables a semantic similarity search across all data (vectors) contained within. +Embeddings are created by sending data to an embedding model, where it is transformed into a vector, which then can be stored as a vector field within its source document in Azure DocumentDB. +Azure DocumentDB supports the creation of vector search indexes on top of these vector fields. +A vector search index is a collection of vectors in [latent space](https://idl.cs.washington.edu/papers/latent-space-cartography/) that enables a semantic similarity search across all data (vectors) contained within. -![A typical embedding pipeline that demonstrates how source data is transformed into vectors using an embedding model then stored in a document in an Azure Cosmos DB vCore database and exposed via a vector search index.](media/embedding_pipeline.png) +![A typical embedding pipeline that demonstrates how source data is transformed into vectors using an embedding model then stored in a document in a Azure DocumentDB database and exposed via a vector search index.](media/embedding_pipeline.png) ## Why vector search? -Vector search is an important RAG (Retrieval Augmented Generation) pattern component. Large Language Model (LLM) data is trained on a snapshot of public data at a point in time. This data does not contain recent public information, nor does it collect private, corporate information. LLMs are also very broad in their knowledge, and including information from a RAG process can help it focus accurately on a specific domain. +Vector search is an important RAG (Retrieval Augmented Generation) pattern component. +Large Language Model (LLM) data is trained on a snapshot of public data at a point in time. +This data does not contain recent public information, nor does it collect private, corporate information. +LLMs are also very broad in their knowledge, and including information from a RAG process can help it focus accurately on a specific domain. -A vector index search allows for a prompt pre-processing step where relevant information can be semantically retrieved from an index and then used to generate a factually accurate prompt for the LLM to reason over. This provides the knowledge augmentation and focus (attention) to the LLM. +A vector index search allows for a prompt pre-processing step where relevant information can be semantically retrieved from an index and then used to generate a factually accurate prompt for the LLM to reason over. +This provides the knowledge augmentation and focus (attention) to the LLM. -In this example, assume textual data is vectorized and stored within an vCore-based Azure Cosmos DB for MongoDB database. The text data and embeddings/vector field are stored in the same document. A vector search index has been created on the vector field. When a message is received from a chat application, this message is also vectorized using the same embedding model (ex., Azure OpenAI text-embedding-ada-002), which is then used as input to the vector search index. The vector search index returns a list of documents whose vector field is semantically similar to the incoming message. The unvectorized text stored within the same document is then used to augment the LLM prompt. The LLM receives the prompt and responds to the requestor based on the information it has been given. +In this example, assume textual data is vectorized and stored within an Azure DocumentDB database. +The text data and embeddings/vector field are stored in the same document. +A vector search index has been created on the vector field. +When a message is received from a chat application, this message is also vectorized using the same embedding model (ex., Azure OpenAI text-embedding-ada-002), which is then used as input to the vector search index. +The vector search index returns a list of documents whose vector field is semantically similar to the incoming message. +The unvectorized text stored within the same document is then used to augment the LLM prompt. +The LLM receives the prompt and responds to the requestor based on the information it has been given. -![A typical vector search request in a RAG scenario depicts an incoming message getting vectorized and used as input to a vector store index search. Multiple results of the vector search are used to build a prompt fed to the LLM. The LLM returns a response to the requestor.](media/vector_search_flow.png) +![A typical vector search request in a RAG scenario depicts an incoming message getting vectorized and used as input to a vector store index search. +Multiple results of the vector search are used to build a prompt fed to the LLM. +The LLM returns a response to the requestor.](media/vector_search_flow.png) -## Why use vCore-based Azure Cosmos DB for MongoDB as a vector store? +## Why use Azure DocumentDB as a vector store? -It is common practice to store vectorized data in a dedicated vector store as vector search indexing is not a common capability of most databases. However, this introduces additional complexity to the solution as the data must be stored in two different locations. vCore-based Azure Cosmos DB for MongoDB supports vector search indexing, which means that the vectorized data can be stored in the same document as the original data. This reduces the complexity of the solution and allows for a single database to be used for both the vector store and the original data. +It is common practice to store vectorized data in a dedicated vector store as vector search indexing is not a common capability of most databases. +However, this introduces additional complexity to the solution as the data must be stored in two different locations. +Azure DocumentDB supports vector search indexing, which means that the vectorized data can be stored in the same document as the original data. +This reduces the complexity of the solution and allows for a single database to be used for both the vector store and the original data. -## Lab - Use vector search on embeddings in vCore-based Azure Cosmos DB for MongoDB +## Lab - Use vector search on embeddings in Azure DocumentDB -In this lab, we'll demonstrate how to add an embedding field to a document, create a vector search index, and perform a vector search query. The lab ends with a demonstration of utilizing vector search with an LLM in a RAG scenario using Azure OpenAI. +In this lab, we'll demonstrate how to add an embedding field to a document, create a vector search index, and perform a vector search query. +The lab ends with a demonstration of utilizing vector search with an LLM in a RAG scenario using Azure OpenAI. -This lab requires the Azure OpenAI endpoint and access key to be added to the settings (`.env`) file. Access this information by opening [Azure OpenAI Studio](https://oai.azure.com/portal) and selecting the **Gear**/Settings icon located to the right in the top toolbar. +This lab requires the Azure OpenAI endpoint and access key to be added to the settings (`.env`) file. +Access this information by opening [Azure OpenAI Studio](https://oai.azure.com/portal) and selecting the **Gear**/Settings icon located to the right in the top toolbar. ![Azure OpenAI Studio displays with the Gear icon highlighted in the top toolbar.](media/azure_openai_studio_settings_icon.png) @@ -36,11 +59,13 @@ On the **Settings** screen, select the **Resource** tab, then copy and record th ![The Azure OpenAI resource settings screen displays with the endpoint and key values highlighted.](media/azure_openai_settings.png) ->**NOTE**: This lab can only be completed using a deployed vCore-based Azure Cosmos DB for MongoDB account due to the use of vector search. The Azure Cosmos DB Emulator does not support vector search. +> **NOTE**: This lab can only be completed using a deployed Azure DocumentDB account due to the use of vector search. +> The Azure Cosmos DB Emulator does not support vector search. -This lab also requires the data provided in the previous lab titled [Load data into Azure Cosmos DB API for MongoDB collections](../08_Load_Data/README.md#lab---load-data-into-azure-cosmos-db-api-for-mongodb-collections). Run all cells in this notebook to prepare the data for use in this lab. +This lab also requires the data provided in the previous lab titled [Load data into Azure Cosmos DB for MongoDB collections](../08_Load_Data/README.md#lab---load-data-into-azure-cosmos-db-api-for-mongodb-collections). +Run all cells in this notebook to prepare the data for use in this lab. -Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/vector_search/README.md). +Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/lab_3_vector_search/README.md). Some highlights from the lab include: @@ -51,9 +76,9 @@ const { OpenAIClient, AzureKeyCredential } = require("@azure/openai"); // Instantiate an AzureOpenAI client const ai_client = new OpenAIClient( - AOAI_ENDPOINT, - new AzureKeyCredential(AOAI_KEY) - ) + AOAI_ENDPOINT, + new AzureKeyCredential(AOAI_KEY) +); ``` ### Vectorizing text using Azure OpenAI @@ -61,75 +86,98 @@ const ai_client = new OpenAIClient( ```javascript //Generate embedding vectors from a text string async function generateEmbeddings(text) { - const embeddings = await aoaiClient.getEmbeddings(embeddingsDeploymentName, text); - // Rest period to avoid rate limiting on Azure OpenAI - await new Promise(resolve => setTimeout(resolve, 500)); - return embeddings.data[0].embedding; + const embeddings = await aoaiClient.getEmbeddings( + embeddingsDeploymentName, + text + ); + // Rest period to avoid rate limiting on Azure OpenAI + await new Promise((resolve) => setTimeout(resolve, 500)); + return embeddings.data[0].embedding; } ``` -### Adding an embedding field to a document and creating a vector search index on a collection +### Adding an embedding field to a document The lab creates an embedding field named `contentVector` in each collection and populates the value with the vectorized text of the JSON representation of the document. ```javascript async function addCollectionContentVectorField(db, collectionName) { - const collection = db.collection(collectionName); - const docs = await collection.find({}).toArray(); - const bulkOperations = []; - console.log(`Generating content vectors for ${docs.length} documents in ${collectionName} collection`); - for (let i=0; i 0) { - console.log(`Persisting the generated content vectors in the ${collectionName} collection using bulkWrite upserts`); - await collection.bulkWrite(bulkOperations); - console.log(`Finished persisting the content vectors to the ${collectionName} collection`); + const content = JSON.stringify(doc); + const contentVector = await generateEmbeddings(content); + bulkOperations.push({ + updateOne: { + filter: { _id: doc["_id"] }, + update: { $set: { contentVector: contentVector } }, + upsert: true, + }, + }); + //output progress every 25 documents + if ((i + 1) % 25 === 0 || i === docs.length - 1) { + console.log( + `Generated ${i + 1} content vectors of ${ + docs.length + } in the ${collectionName} collection` + ); } + } + if (bulkOperations.length > 0) { + console.log( + `Persisting the generated content vectors in the ${collectionName} collection using bulkWrite upserts` + ); + await collection.bulkWrite(bulkOperations); + console.log( + `Finished persisting the content vectors to the ${collectionName} collection` + ); + } +``` - //check to see if the vector index already exists on the collection - console.log(`Checking if vector index exists in the ${collectionName} collection`) - const vectorIndexExists = await collection.indexExists('VectorSearchIndex'); - if (!vectorIndexExists) { - await db.command({ - "createIndexes": collectionName, - "indexes": [ - { - "name": "VectorSearchIndex", - "key": { - "contentVector": "cosmosSearch" - }, - "cosmosSearchOptions": { - "kind": "vector-ivf", - "numLists": 1, - "similarity": "COS", - "dimensions": 1536 - } - } - ] - }); - console.log(`Created vector index on contentVector field on ${collectionName} collection`); - } - else { - console.log(`Vector index already exists on contentVector field in the ${collectionName} collection`); - } +### Creating a vector search index + +Enabling vector search on the `contentVector` field in the collection. + +```javascript +//check to see if the vector index already exists on the collection +console.log( + `Checking if vector index exists in the ${collectionName} collection` +); +const vectorIndexExists = await collection.indexExists("VectorSearchIndex"); +if (!vectorIndexExists) { + await db.command({ + createIndexes: collectionName, + indexes: [ + { + name: "VectorSearchIndex", + key: { + contentVector: "cosmosSearch", + }, + cosmosSearchOptions: { + kind: "vector-ivf", + numLists: 1, + similarity: "COS", + dimensions: 1536, + }, + }, + ], + }); + console.log( + `Created vector index on contentVector field on ${collectionName} collection` + ); +} else { + console.log( + `Vector index already exists on contentVector field in the ${collectionName} collection` + ); } ``` @@ -137,75 +185,88 @@ async function addCollectionContentVectorField(db, collectionName) { ```javascript async function vectorSearch(db, collectionName, query, numResults = 3) { - const collection = db.collection(collectionName); - // generate the embedding for incoming question - const queryEmbedding = await generateEmbeddings(query); - - const pipeline = [ - { - '$search': { - "cosmosSearch": { - "vector": queryEmbedding, - "path": "contentVector", - "k": numResults - }, - "returnStoredSource": true - } + const collection = db.collection(collectionName); + // generate the embedding for incoming question + const queryEmbedding = await generateEmbeddings(query); + + const pipeline = [ + { + $search: { + cosmosSearch: { + vector: queryEmbedding, + path: "contentVector", + k: numResults, }, - { '$project': { 'similarityScore': { '$meta': 'searchScore' }, 'document': '$$ROOT' } } - ]; - - //perform vector search and return the results as an array - const results = await collection.aggregate(pipeline).toArray(); - return results; + returnStoredSource: true, + }, + }, + { + $project: { + similarityScore: { $meta: "searchScore" }, + document: "$$ROOT", + }, + }, + ]; + + //perform vector search and return the results as an array + const results = await collection.aggregate(pipeline).toArray(); + return results; } ``` ### Using vector search results with an LLM in a RAG scenario ```javascript -async function ragWithVectorsearch(db, collectionName, question, numResults=3) { - //A system prompt describes the responsibilities, instructions, and persona of the AI. - const systemPrompt = ` +async function ragWithVectorSearch( + db, + collectionName, + question, + numResults = 3 +) { + // A system prompt describes the responsibilities, instructions, and persona of the AI. + const systemPrompt = ` You are a helpful, fun and friendly sales assistant for Cosmic Works, a bicycle and bicycle accessories store. Your name is Cosmo. You are designed to answer questions about the products that Cosmic Works sells. - + Only answer questions related to the information provided in the list of products below that are represented in JSON format. - + If you are asked a question that is not in the list, respond with "I don't know." - + List of products: `; - const collection = db.collection(collectionName); - //generate vector embeddings for the incoming question - const queryEmbedding = await generateEmbeddings(question); - //perform vector search and return the results - results = await vectorSearch(db, collectionName, question, numResults); - productList = ""; - //remove contentVector from the results, create a string of the results for the prompt - for (const result of results) { - delete result['document']['contentVector']; - productList += JSON.stringify(result['document']) + "\n\n"; - } - - //assemble the prompt for the large language model (LLM) - const formattedPrompt = systemPrompt + productList; - //prepare messages for the LLM call, TODO: if message history is desired, add them to this messages array - const messages = [ - { - "role": "system", - "content": formattedPrompt - }, - { - "role": "user", - "content": question - } - ]; - - //call the Azure OpenAI model to get the completion and return the response - const completion = await aoaiClient.getChatCompletions(completionsDeploymentName, messages); - return completion.choices[0].message.content; + const collection = db.collection(collectionName); + // generate vector embeddings for the incoming question + const queryEmbedding = await generateEmbeddings(question); + // perform vector search and return the results + results = await vectorSearch(db, collectionName, question, numResults); + productList = ""; + // remove contentVector from the results, create a string of the results for the prompt + for (const result of results) { + delete result["document"]["contentVector"]; + productList += JSON.stringify(result["document"]) + "\n\n"; + } + + // assemble the prompt for the large language model (LLM) + const formattedPrompt = systemPrompt + productList; + // prepare messages for the LLM call, if message history is desired, add them to this messages array + const messages = [ + { + role: "system", + content: formattedPrompt, + }, + { + role: "user", + content: question, + }, + ]; + + // call the Azure OpenAI model to get the completion and return the response + const completion = await aoaiClient.getChatCompletions( + completionsDeploymentName, + messages + ); + return completion.choices[0].message.content; } ``` diff --git a/10_LangChain/README.md b/10_LangChain/README.md index 10667f7..06be491 100644 --- a/10_LangChain/README.md +++ b/10_LangChain/README.md @@ -1,28 +1,44 @@ # LangChain -[LangChain](https://www.langchain.com/) is an open-source framework designed to simplify the creation of applications that use large language models (LLMs). LangChain has a vibrant community of developers and contributors and is used by many companies and organizations. LangChain utilizes proven Prompt Engineering patterns and techniques to optimize LLMs, ensuring successful and accurate results through verified and tested best practices. +[LangChain](https://www.langchain.com/) is an open-source framework designed to simplify the creation of applications that use large language models (LLMs). +LangChain has a vibrant community of developers and contributors and is used by many companies and organizations. +LangChain utilizes proven Prompt Engineering patterns and techniques to optimize LLMs, ensuring successful and accurate results through verified and tested best practices. -Part of the appeal of LangChain syntax is the capability of breaking down large complex interactions with LLMs into smaller, more manageable steps by composing a reusable [chain](https://python.langchain.com/docs/modules/chains/) process. LangChain provides a syntax for chains([LCEL](https://python.langchain.com/docs/modules/chains/#lcel)), the ability to integrate with external systems through [tools](https://python.langchain.com/docs/integrations/tools/), and end-to-end [agents](https://python.langchain.com/docs/modules/agents/) for common applications. +Part of the appeal of LangChain syntax is the capability of breaking down large complex interactions with LLMs into smaller, more manageable steps which can be linked together to form "chains". +In addition to chains, LangChain provides an ability to integrate with external systems through [tools](https://docs.langchain.com/oss/javascript/langchain/tools), and end-to-end [agents](https://docs.langchain.com/oss/javascript/langchain/agents) for common applications. -The concept of an agent is quite similar to that of a chain in LangChain but with one fundamental difference. A chain in LangChain is a hard-coded sequence of steps executed in a specific order. Conversely, an agent leverages the LLM to assess the incoming request with the current context to decide what steps or actions need to be executed and in what order. +The concept of an agent is quite similar to that of a chain in LangChain but with one fundamental difference. +A chain in LangChain is a hard-coded sequence of steps executed in a specific order. +Conversely, an agent leverages the LLM to assess the incoming request with the current context to decide what steps or actions need to be executed and in what order. -LangChain agents can leverage tools and toolkits. A tool can be an integration into an external system, custom code, or even another chain. A toolkit is a collection of tools that can be used to solve a specific problem. +LangChain agents can leverage tools and toolkits. +A tool can be an integration into an external system, custom code, a retriever, or even another chain. +A toolkit is a collection of tools that can be used to solve a specific problem. ## LangChain RAG pattern -Earlier in this guide, the RAG (Retrieval Augmented Generation) pattern was introduced. In LangChain, the RAG pattern is implemented as part of a chain that combines a retriever and a Large Language Model (generator). The retriever is responsible for finding the most relevant documents for a given query, in this case, doing a vector search on vCore-based Azure Cosmos DB for MongoDB, and the LLM (generator) is responsible for reasoning over the incoming prompt and context. +Earlier in this guide, the RAG (Retrieval Augmented Generation) pattern was introduced. +In LangChain, the RAG pattern is implemented as part of a chain that combines a retriever and a Large Language Model (generator). +The retriever is responsible for finding the most relevant documents for a given query, in this case, doing a vector search on Azure DocumentDB, and the LLM (generator) is responsible for reasoning over the incoming prompt and context. ![LangChain RAG diagram shows the flow of an incoming message through a retriever, augmenting the prompt, parsing the output and returning the final message.](media/langchain_rag.png) -When an incoming message is received, the retriever will vectorize the message and perform a vector search to find the most relevant documents for the given query. The retriever returns a list of documents that are then used to augment the prompt. The augmented prompt is then passed to the LLM (generator) to reason over the prompt and context. The output from the LLM is then parsed and returned as the final message. +When an incoming message is received, the retriever will vectorize the message and perform a vector search to find the most relevant documents for the given query. +The retriever returns a list of documents that are then used to augment the prompt. +The augmented prompt is then passed to the LLM (generator) to reason over the prompt and context. +The output from the LLM is then parsed and returned as the final message. -> **Note**: A vector store retriever is only one type of retriever that can be used in the RAG pattern. Learn more about retrievers in the [LangChain documentation](https://python.langchain.com/docs/modules/data_connection/retrievers/). +> **Note**: A vector store retriever is only one type of retriever that can be used in the RAG pattern. + +Learn more about retrievers in the [LangChain retriever documentation](https://reference.langchain.com/javascript/langchain_core/retrievers/). ## Lab - Vector search and RAG using LangChain -In this lab uses LangChain to re-implement the RAG pattern introduced in the previous lab. Take note of the readability of the code and how easy it is to compose a reusable RAG chain using LangChain that queries the products vector index in vCore-based Azure Cosmos DB for MongoDB. The lab concludes with the creation of an agent with various tools for the LLM to leverage to fulfill the incoming request. +In this lab uses LangChain to re-implement the RAG pattern introduced in the previous lab. +Take note of the readability of the code and how easy it is to compose a reusable RAG chain using LangChain that queries the products vector index in Azure DocumentDB. +The lab concludes with the creation of an agent with various tools for the LLM to leverage to fulfill the incoming request. -Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/langchain/README.md). +Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/lab_4_langchain/README.md). Some highlights of the lab include: @@ -34,125 +50,132 @@ const dbClient = new MongoClient(process.env.AZURE_COSMOSDB_CONNECTION_STRING); // set up the Azure Cosmos DB vector store const azureCosmosDBConfig = { - client: dbClient, - databaseName: "cosmic_works", - collectionName: "products", - indexName: "VectorSearchIndex", - embeddingKey: "contentVector", - textKey: "_id" -} -const vectorStore = new AzureCosmosDBVectorStore(new OpenAIEmbeddings(), azureCosmosDBConfig); + client: dbClient, + databaseName: "cosmic_works", + collectionName: "products", + indexName: "VectorSearchIndex", + embeddingKey: "contentVector", + textKey: "_id", +}; +const vectorStore = new AzureCosmosDBMongoDBVectorStore( + new OpenAIEmbeddings(), + azureCosmosDBConfig +); ``` ### Composing a reusable RAG chain ```javascript async function ragLCELChain(question) { - // A system prompt describes the responsibilities, instructions, and persona of the AI. - // Note the addition of the templated variable/placeholder for the list of products and the incoming question. - const systemPrompt = ` - You are a helpful, fun and friendly sales assistant for Cosmic Works, a bicycle and bicycle accessories store. - Your name is Cosmo. - You are designed to answer questions about the products that Cosmic Works sells. - - Only answer questions related to the information provided in the list of products below that are represented - in JSON format. - - If you are asked a question that is not in the list, respond with "I don't know." - - Only answer questions related to Cosmic Works products, customers, and sales orders. - - If a question is not related to Cosmic Works products, customers, or sales orders, - respond with "I only answer questions about Cosmic Works" - - List of products: - {products} - - Question: - {question} - `; - const retriever = vectorStore.asRetriever(); - const prompt = PromptTemplate.fromTemplate(systemPrompt); - - // The RAG chain will populate the variable placeholders of the system prompt - // with the formatted list of products based on the documents retrieved from the vector store. - // The RAG chain will then invoke the LLM with the populated prompt and the question. - // The response from the LLM is then parsed as a string and returned. - const ragChain = RunnableSequence.from([ - { - products: retriever.pipe(formatDocuments), - question: new RunnablePassthrough() - }, - prompt, - chatModel, - new StringOutputParser() - ]); - - return await ragChain.invoke(question); + // A system prompt describes the responsibilities, instructions, and persona of the AI. + // Note the addition of the templated variable/placeholder for the list of products and the incoming question. + const systemPrompt = ` + You are a helpful, fun and friendly sales assistant for Cosmic Works, a bicycle and bicycle accessories store. + Your name is Cosmo. + You are designed to answer questions about the products that Cosmic Works sells. + + Only answer questions related to the information provided in the list of products below that are represented + in JSON format. + + If you are asked a question that is not in the list, respond with "I don't know." + + Only answer questions related to Cosmic Works products, customers, and sales orders. + + If a question is not related to Cosmic Works products, customers, or sales orders, + respond with "I only answer questions about Cosmic Works" + + List of products: + {products} + + Question: + {question} + `; + const retriever = vectorStore.asRetriever(); + const prompt = PromptTemplate.fromTemplate(systemPrompt); + + // The RAG chain will populate the variable placeholders of the system prompt + // with the formatted list of products based on the documents retrieved from the vector store. + // The RAG chain will then invoke the LLM with the populated prompt and the question. + // The response from the LLM is then parsed as a string and returned. + const ragChain = RunnableSequence.from([ + { + products: retriever.pipe(formatDocuments), + question: new RunnablePassthrough(), + }, + prompt, + chatModel, + new StringOutputParser(), + ]); + + return await ragChain.invoke(question); } ``` ### Creating tools for LangChain agents to use -Tools are selected by the Large Language model at runtime. In this case, depending on the incoming user request the LLM will decide which collection in the database to query. The following code shows how to create a tool for the LLM to use to query the products collection in the database. +Tools are selected by the Large Language model at runtime. +In this case, depending on the incoming user request the LLM will decide which collection in the database to query. +The following code shows how to create a tool for the LLM to use to query the products collection in the database. ```javascript // A tool that retrieves product information from Cosmic Works based on the user's question. const productsRetrieverTool = new DynamicTool({ - name: "products_retriever_tool", - description: `Searches Cosmic Works product information for similar products based on the question. - Returns the product information in JSON format.`, - func: async (input) => await retrieverChain.invoke(input), + name: "products_retriever_tool", + description: `Searches Cosmic Works product information for similar products based on the question. + Returns the product information in JSON format.`, + func: async (input) => await retrieverChain.invoke(input), }); ``` -### Creating tools that call Python functions +### Creating tools that call JavaScript functions -Users may query for information that does not have a semantic meaning, such as an ID GUID value or a SKU number. Providing agents with tools to call Python functions to retrieve documents based on these fields is a common practice. The following is an example of adding tools that call out to Python functions for the products collection. +Users may query for information that does not have a semantic meaning, such as an ID GUID value or a SKU number. +Providing agents with tools to call JavaScript functions to retrieve documents based on these fields is a common practice. +The following is an example of adding tools that call out to JavaScript functions for the products collection. ```javascript // A tool that will lookup a product by its SKU. Note that this is not a vector store lookup. const productLookupTool = new DynamicTool({ - name: "product_sku_lookup_tool", - description: `Searches Cosmic Works product information for a single product by its SKU. - Returns the product information in JSON format. - If the product is not found, returns null.`, - func: async (input) => { - const db = dbClient.db("cosmic_works"); - const products = db.collection("products"); - const doc = await products.findOne({ "sku": input }); - if (doc) { - //remove the contentVector property to save on tokens - delete doc.contentVector; - } - return doc ? JSON.stringify(doc, null, '\t') : null; - }, + name: "product_sku_lookup_tool", + description: `Searches Cosmic Works product information for a single product by its SKU. + Returns the product information in JSON format. + If the product is not found, returns null.`, + func: async (input) => { + const db = dbClient.db("cosmic_works"); + const products = db.collection("products"); + const doc = await products.findOne({ sku: input }); + if (doc) { + //remove the contentVector property to save on tokens + delete doc.contentVector; + } + return doc ? JSON.stringify(doc, null, "\t") : null; + }, }); ``` -### Creating an agent armed with tools for vector search and Python functions calling +### Creating an agent armed with tools for vector search and JavaScript functions calling ```javascript // Define the agent and executor // An agent is a type of chain that reasons over the input prompt and has the ability // to decide which function(s) (tools) to use and parses the output of the functions. -const runnableAgent = RunnableSequence.from([ - { - input: (i) => i.input, - agent_scratchpad: (i) => formatToOpenAIFunctionMessages(i.steps), - }, - prompt, - modelWithFunctions, - new OpenAIFunctionsAgentOutputParser(), +const runnableAgent = RunnableSequence.from([ + { + input: (i) => i.input, + agent_scratchpad: (i) => formatToOpenAIFunctionMessages(i.steps), + }, + prompt, + modelWithFunctions, + new OpenAIFunctionsAgentOutputParser(), ]); // An agent executor can be thought of as a runtime, it orchestrates the actions of the agent // until completed. This can be the result of a single or multiple actions (one can feed into the next). -// Note: If you wish to see verbose output of the tool usage of the agent, +// Note: If you wish to see verbose output of the tool usage of the agent, // set returnIntermediateSteps to true const executor = AgentExecutor.fromAgentAndTools({ - agent: runnableAgent, - tools, - returnIntermediateSteps: true + agent: runnableAgent, + tools, + returnIntermediateSteps: true, }); ``` diff --git a/11_Backend_API/README.md b/11_Backend_API/README.md index 275c091..1ef716f 100644 --- a/11_Backend_API/README.md +++ b/11_Backend_API/README.md @@ -1,9 +1,9 @@ # Lab - Backend API -In the previous lab, a LangChain agent was created armed with tools to do vector lookups and concrete document id lookups via function calling. In this lab, the agent functionality needs to be extracted into a backend api for the frontend application that will allow users to interact with the agent. +In the previous lab, a LangChain agent was created armed with tools to do vector lookups and concrete document id lookups via function calling. +In this lab, the agent functionality needs to be extracted into a backend api for the frontend application that will allow users to interact with the agent. -This lab implements a backend API using FastAPI that exposes the LangChain agent functionality. The provided code leverages Docker containers and includes full step-by-step instructions to run and test the API locally as well as deployed to [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/overview) (leveraging the [Azure Container Registry](https://learn.microsoft.com/azure/container-registry/)). +This lab implements a backend API using FastAPI that exposes the LangChain agent functionality. +The provided code leverages Docker containers and includes full step-by-step instructions to run and test the API locally as well as deployed to [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/overview) (leveraging the [Azure Container Registry](https://learn.microsoft.com/azure/container-registry/)). -This lab also requires the data provided in the previous lab titled [Load data into Azure Cosmos DB API for MongoDB collections](../08_Load_Data/README.md#lab---load-data-into-azure-cosmos-db-api-for-mongodb-collections) as well as the populated vector index created in the lab titled [Vector Search using vCore-based Azure Cosmos DB for MongoDB](../09_Vector_Search_Cosmos_DB/README.md#lab---use-vector-search-on-embeddings-in-vcore-based-azure-cosmos-db-for-mongodb). Run all cells in both notebooks to prepare the data for use in this lab. - -Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/backend_api/README.md). +Please visit the lab repository to complete [this lab](https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide/blob/main/Labs/lab_5_backend_api/README.md). diff --git a/12_User_Interface/README.md b/12_User_Interface/README.md index ddcea08..573b3c0 100644 --- a/12_User_Interface/README.md +++ b/12_User_Interface/README.md @@ -1,6 +1,8 @@ # Connect the chat user interface with the chatbot API -In the previous lab, the backend API code was configured and deployed. The backend API integrates vCore-based Azure Cosmos DB for MongoDB with Azure OpenAI. When the Azure resource template for this lab was run to deploy the necessary Azure resources, a front-end web application written as a SPA (single page application) in React was deployed. +In the previous lab, the backend API code was configured and deployed. +The backend API integrates Azure DocumentDB with Azure OpenAI. +When the Azure resource template for this lab was run to deploy the necessary Azure resources, a front-end web application written as a SPA (single page application) in React was deployed. The URL to access this front-end application within the Azure Portal on the **Web App** resource with the name that ends with **-web**. @@ -8,28 +10,38 @@ The following screenshot shows where to find the front-end application URL: ![Web App resource for front-end application with Default domain highlighted](images/2024-01-17-12-41-48.png) -Navigating to this URL in the browser accesses the front-end application. Through this front-end application User Interface, questions can be submitted to the Azure OpenAI model about the CosmicWorks company data, then it will generate responses accordingly. +Navigating to this URL in the browser accesses the front-end application. +Through this front-end application User Interface, questions can be submitted to the Azure OpenAI model about the CosmicWorks company data, then it will generate responses accordingly. ![Front-end Web Application User Interface](images/2024-01-17-12-42-59.png) -While the code for the SPA web application is outside the scope of this dev guide. It's worth noting that the Web App is configured with the URL for the Backend API using the **Application setting** named `API_ENDPOINT`. When the application was deployed as part of the Azure template deployment, it was automatically configured with this URL to connect the front-end SPA web application to the Backend API. +While the code for the SPA web application is outside the scope of this dev guide. +It's worth noting that the Web App is configured with the URL for the Backend API using the **Application setting** named `API_ENDPOINT`. +When the application was deployed as part of the Azure template deployment, it was automatically configured with this URL to connect the front-end SPA web application to the Backend API. ![Web App resource showing the application settings with the API_ENDPOINT setting highlighted](images/2024-01-17-12-45-30.png) ## Ask questions about data and observe the responses -To ask the AI questions about the CosmicWorks company data, type the questions in to the front-end application chat user interface. The web application includes tiles with a couple example questions to get started. To use these, simply click on the question tile and it will generate an answer. +To ask the AI questions about the CosmicWorks company data, type the questions in to the front-end application chat user interface. +The web application includes tiles with a couple example questions to get started. +To use these, simply click on the question tile and it will generate an answer. ![Front-end Web Application User Interface](images/2024-01-17-12-42-59.png) These example questions are: + - What was the price of the product with sku `FR-R92B-58`? - What is the SKU of HL Road Frame - Black? - What is HL Road Frame? -> **Note**: It's possible the first time you ask a question within the Front end application there may be an error. Occasionally when the Azure Bicep template deploys the front end application there will be an issue configuring the use of the `API_ENDPOINT` app setting. If this happens, simply navigate to **Deployment** -> **Deployment Center**, then click **Sync** to have the Web App refresh the deployment of the front end app from it's GitHub repository source code. This should fix that error. +> **Note**: It's possible the first time you ask a question within the Front end application there may be an error. +Occasionally when the Azure Bicep template deploys the front end application there will be an issue configuring the use of the `API_ENDPOINT` app setting. +If this happens, simply navigate to **Deployment** -> **Deployment Center**, then click **Sync** to have the Web App refresh the deployment of the front end app from it's GitHub repository source code. +This should fix that error. The chat user interface presents as a traditional chat application style interface when asking questions. +It also includes a "New chat" button to open new chat sessions, and the list of previous chat sessions on the left side of the UI that enables you to toggle between sessions. ![Chat user interface screenshot with question and generated answer displayed](images/2024-01-17-12-53-13.png) @@ -37,51 +49,62 @@ Go ahead, ask the service a few questions about CosmicWorks and observe the resp ## What do I do if the responses are incorrect? -It's important to remember the model is pre-trained with data, given a system message to guide it, in addition to the company data it has access to via vCore-based Azure Cosmos DB for MongoDB. There are times when the Azure OpenAI model may generate an incorrect response to the prompt given that is either incomplete or even a hallucination (aka includes information that is not correct or accurate). +It's important to remember the model is pre-trained with data, given a system message to guide it, in addition to the company data it has access to via Azure DocumentDB. +There are times when the Azure OpenAI model may generate an incorrect response to the prompt given that is either incomplete or even a hallucination (aka includes information that is not correct or accurate). There are a few options of how this can be handled when the response is incorrect: -1. Provide a new prompt that includes more specific and structured information that can help the model generate a more accurate response. -2. Include more data in the library of company information the model has access to. The incorrect response may be a result of data or information about the company that is missing currently. -3. Use Prompt Engineering techniques to enhance the System message and/or Supporting information provided to guide the model. +1. Provide a new prompt that includes more specific and structured information that can help the model generate a more accurate response. +2. Include more data in the library of company information the model has access to. + The incorrect response may be a result of data or information about the company that is missing currently. +3. Use Prompt Engineering techniques to enhance the System message and/or Supporting information provided to guide the model. While it may be simple to ask the model questions, there are times when Prompt Engineering skills may be necessary to get the most value and reliable responses from the AI model. ## What happens when I start exceeding my token limits? -A Token in Azure OpenAI is a basic unit of input and output that the service processes. Generally, the models understand and process text by breaking it down into tokens. +A Token in Azure OpenAI is a basic unit of input and output that the service processes. +Generally, the models understand and process text by breaking it down into tokens. -For example, the word `hamburger` gets broken up into the tokens `ham`, `bur` and `ger`, while a short and common word like `pear` is a single token. Many tokens start with a whitespace, for example ` hello` and ` bye`. +For example, the word `hamburger` gets broken up into the tokens `ham`, `bur` and `ger`, while a short and common word like `pear` is a single token. +Many tokens start with a whitespace, for example ` hello` and ` bye`. -The total number of tokens processed in a given request depends on the length of the input, output and request parameters. The quantity of tokens being processed will also affect the response latency and throughput for the models. +The total number of tokens processed in a given request depends on the length of the input, output and request parameters. +The quantity of tokens being processed will also affect the response latency and throughput for the models. -> **Note**: The [pricing of the Azure OpenAI](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) service is primarily based on token usage. +> **Note**: The [pricing of Azure Open AI](https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/) service is primarily based on token usage. ### Exceeding Token Quota Limits -Azure OpenAI has **tokens per minute** quota limits on the service. This quota limit, is based on the OpenAI model being used and the Azure region it's hosted in. +Azure OpenAI has **tokens per minute** quota limits on the service. +This quota limit, is based on the OpenAI model being used and the Azure region it's hosted in. -> **Note**: The [Azure OpenAI Quotas and limits documentation](https://learn.microsoft.com/azure/ai-services/openai/quotas-limits) contains further information on the specific quotas per OpenAI model and Azure region. +> **Note**: The [Azure Open AI quotas and limits](https://learn.microsoft.com/azure/ai-foundry/openai/quotas-limits) contains further information on the specific quotas per OpenAI model and Azure region. If an applications usage of an Azure OpenAI model exceeds the token quota limits, then the service will respond with a **Rate Limit Error** (Error code 429). When this error is encountered, there are a couple options available for handling it: - **Wait a minute** - With the tokens quota limit being a rate limits of the maximum number of tokens allowed per minute, the application will be able to send more prompts to the model again after the quota resets each minute. -- **Request a quota increase** - It may be possible to get Microsoft to increase the token quota to a higher limit, but it's not guaranteed to be approved. This request can be made at [https://aka.ms/oai/quotaincrease](https://aka.ms/oai/quotaincrease) +- **Request a quota increase** - It may be possible to get Microsoft to increase the token quota to a higher limit, but it's not guaranteed to be approved. +This request can be made at [AI Foundry Request for Quota Increase](https://aka.ms/oai/quotaincrease) ### Tips to Minimize Token Rate Limit Errors Here are a few tips that can help to minimize an applications token rate limit errors: -- **Retry Logic** - Implement retry logic in the application so it will retry the call to the Azure OpenAI model, rather than throwing an exception the first time. This is generally best practices when consuming external APIs from applications so they can gracefully handle any unexpected exceptions. -- **Scale Workload Gradually** - Avoid increasing the workload of the application too quickly. By gradually increasing the scale of the workload. -- **Asynchronous Load Patterns** - While there are time sensitive operations that will require a response immediately, there are also operations that are able to be run more asynchronously. Background processes or other similar operations could be build in a way to perform a combination of rate limiting the applications own usage of the model, or even delaying calls until periods of the day where the application is under less load. +- **Retry Logic** - Implement retry logic in the application so it will retry the call to the Azure OpenAI model, rather than throwing an exception the first time. +This is generally best practices when consuming external APIs from applications so they can gracefully handle any unexpected exceptions. +- **Scale Workload Gradually** - Avoid increasing the workload of the application too quickly. +By gradually increasing the scale of the workload. +- **Asynchronous Load Patterns** - While there are time sensitive operations that will require a response immediately, there are also operations that are able to be run more asynchronously. +Background processes or other similar operations could be build in a way to perform a combination of rate limiting the applications own usage of the model, or even delaying calls until periods of the day where the application is under less load. - **Set `max_tokens`** - Setting a lower `max_tokens` when calling the service when a short response is expected will limit the maximum number of tokens allowed for the generated answer. - **Set `best_of`** - Setting a lower `best_of` when calling the service enables the application to control the number of candidate completions generated and how many to return from the service. ### Exceeding Token Limit for System message -When configuring a System message to guide the generated responses, there is a limit on how long the System message can be. The token limit for the System message is 400 tokens. +When configuring a System message to guide the generated responses, there is a limit on how long the System message can be. +The token limit for the System message is 400 tokens. If the System message provided is more than 400 tokens, the rest of the tokens beyond the first 400 will be ignored. diff --git a/13_Conclusion/README.md b/13_Conclusion/README.md index 16129b6..6855429 100644 --- a/13_Conclusion/README.md +++ b/13_Conclusion/README.md @@ -1,14 +1,28 @@ # Conclusion -This guide was designed to provide an insightful journey for Python/MongoDB developers to get started with vCore-based Azure Cosmos DB for MongoDB as it applies to creating exciting AI-enabled applications using existing skills. We hope you found this guide helpful and informative. +This guide has provided a comprehensive walkthrough for creating intelligent solutions that combine Azure DocumentDB with vector search capabilities powered by DiskANN and document retrieval with Azure OpenAI services to build a chat bot experience. +By integrating these technologies, you can efficiently manage both operational data and vectors within a single database, while leveraging Azure OpenAI for advanced document retrieval and natural language understanding. + +The benefits of building a chat bot experience using Azure DocumentDB with vector search capabilities powered by DiskANN and Azure OpenAI services includes: + +- **Unified data and vector management**: Storing both operational data and vectors together in a single database reduces complexity, improves performance, and eliminates the need to synchronize data between multiple databases. +- **No need for synchronization**: By keeping data and vectors in one place, you avoid the overhead of synchronizing two different databases. +- **Flexible schema**: Adapt to changing data structures effortlessly, ensuring your system remains flexible and scalable as your application evolves. +- **Support for latency-sensitive applications**: Azure Cosmos DB is optimized for applications requiring low-latency responses, making it suitable for real-time, interactive use cases. +- **High elasticity and throughput**: Azure Cosmos DB can scale seamlessly to handle high-throughput workloads, making it perfect for applications that need to grow dynamically with demand. +- **Store chat history and vector data**: Easily manage chat histories alongside vector and operational data, making it ideal for chat bot and other interactive applications. + +We hope you found this guide helpful and informative. ## Clean up To clean up the resources created in this guide, delete the `mongo-devguide-rg` resource group in the Azure Portal. -Alternatively, you can use the Azure CLI to delete the resource group. The following command deletes the resource group and all resources within it. The `--no-wait` flag makes the command return immediately, without waiting for the deletion to complete. +Alternatively, you can use the Azure CLI to delete the resource group. +The following command deletes the resource group and all resources within it. +The `--no-wait` flag makes the command return immediately, without waiting for the deletion to complete. ->**Note**: Ensure the Azure CLI session is authenticated using `az login` and the correct subscription is selected using `az account set --subscription `. +> **Note**: Ensure the Azure CLI session is authenticated using `az login` and the correct subscription is selected using `az account set --subscription `. ```powershell az group delete --name mongo-devguide-rg --yes --no-wait diff --git a/Backend/README.md b/Backend/README.md index 3f4bcb2..1307dbc 100644 --- a/Backend/README.md +++ b/Backend/README.md @@ -1 +1 @@ -# Cosmos DB Dev Guide Backend App - Node.js +# Cosmos DB Dev Guide Backend App Node.js diff --git a/Labs/backend_api/README.md b/Labs/backend_api/README.md deleted file mode 100644 index 9e47cb8..0000000 --- a/Labs/backend_api/README.md +++ /dev/null @@ -1,243 +0,0 @@ -# Lab - Backend API - -In the previous lab, a LangChain agent was created armed with tools to do vector lookups and concrete document id lookups via function calling. In this lab, the agent functionality needs to be extracted into a backend api for the frontend application that will allow users to interact with the agent. - -The information provided in this section assumes that the dependent infrastructure is deployed and have completed the previous labs in this dev guide. - -## Overview - -The backend api is a Node.js web application, using Express and Swagger, that will expose endpoints for the frontend application to interact with. The backend api is a containerized application that will be deployed to [Azure Container Apps](https://learn.microsoft.com/en-us/azure/container-apps/overview). - -## Clone the Backend API - -Create a folder to house the repository. Open a terminal and navigate to the folder. Clone the repository, then navigate to the `Backend` folder within the repository. - -```bash -git clone https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide.git - -cd Azure-OpenAI-Node.js-Developer-Guide -cd Backend -``` - -## Run the backend api locally - -When developing a backend api, it is often useful to run the application locally to test and debug. This section outlines how to run the backend api locally while watching the file system for code changes. Any detected changes will automatically restart the backend api. - -1. Open the backend api folder location in VS Code. - -2. Open a **Terminal** window in VS Code (CTRL+`). - -3. Setup the `.env` file. Copy the `.env.example` file to `.env` and update the values. These are the same environment variables used in the previous labs. - - ```bash - cp .env.EXAMPLE .env - ``` - -3. Add the following settings to the `.env` file, populating the MongoDB connection string and replacing the values from the deployed Azure OpenAI service: - - ```bash - AZURE_COSMOSDB_CONNECTION_STRING= - AZURE_OPENAI_API_INSTANCE_NAME= - AZURE_OPENAI_API_KEY= - AZURE_OPENAI_API_DEPLOYMENT_NAME=completions - AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME=embeddings - AZURE_OPENAI_API_VERSION=2023-09-01-preview - ``` - - Replace `` with the MongoDB connection string. Replace `` with the name of the deployed OpenAI service, and `` with the Azure OpenAI API key. Leave all other values untouched. - - >**Note**: The Azure OpenAI service name is not the full endpoint. Only the service name is required. For example, if the endpoint is `https://myservicename.openai.azure.com/`, then the service name is `myservicename`. - -5. Run the following command to install any dependencies: - - ```bash - npm install - ``` - -6. Run the following command to start the backend api. - - ```bash - node --env-file=.env app.js - ``` - - ![The VSCode terminal window displays with the backend API started.](media/local_backend_running_console.png "Local backend api running") - -7. Open a browser and navigate to `http://localhost:4242/docs` to view the Swagger UI. - - ![The Swagger UI displays for the locally running backend api](media/local_backend_swagger_ui.png "Local backend api Swagger UI") - -8. Expand the **GET / Root** endpoint and select **Try it out**. Select **Execute** to send the request. The response should display a status of `ready`. - - ![The Swagger UI displays the GET / Root endpoint reponse that has a status of ready.](media/local_backend_swagger_ui_root_response.png "Local backend api Swagger UI Root response") - -9. Expand the **POST /ai** endpoint and select **Try it out**. In the **Request body** field, enter the following JSON. - - ```json - { - "session_id": "abc123", - "prompt": "What was the price of the product with sku `FR-R92B-58`" - } - ``` - -10. Select **Execute** to send the request. Observe that the response indicates the price as being `$1431.50`. - - ![The Swagger UI displays the POST /ai endpoint reponse that has a status of ready.](media/local_backend_swagger_ui_ai_response.png "Local backend api Swagger UI AI response") - -11. In the Terminal window, press CTRL+C to stop the backend api. - -## Build and run the backend api container locally in Docker Desktop - -When deployed to Azure, the backend api will be running in a container. It is important to test the container locally to ensure it is working as expected. Containers are important because they provide a consistent environment for the application to run in. This consistency allows the application to run the same way in development, test, and production environments - whether they be locally or in the cloud. - -The backend api contains a `Dockerfile` that defines the container image and is used by Docker to build the container image. The Dockerfile contains instructions for Docker to build the container image. The container image is a snapshot of the application and its dependencies. The container image can be thought of an installer for the application to be deployed as needed in any environment. - -The `Dockerfile` for the backend api is shown below. - -```dockerfile -FROM node:21 - -WORKDIR /usr/src/app -COPY package*.json ./ -RUN npm install -COPY . . - -EXPOSE 80 - -CMD [ "node", "--env-file=.env", "app.js", "port=80" ] -``` - -Notice the steps of installing the pip dependencies, and running the **uvicorn** command line similar to what was done in the previous section. - -1. Ensure Docker Desktop is running. - -2. Open a **Terminal** window in VS Code (CTRL+`). - -4. Run the following command to build the container image. Once complete, a message displays the operation has `FINISHED`. - - ```bash - docker build --pull --rm -f "DOCKERFILE" -t devguidenodebackendapi:latest "." - ``` - - ![The VSCode terminal window displays the docker build command and the FINISHED message.](media/local_backend_docker_build.png "Local backend api Docker build") - -5. Lastly, run the container in Docker Desktop using the following command. - - ```bash - docker run -d -p 4242:80 --name devguide-backend-api devguidenodebackendapi:latest - ``` - - ![The VSCode terminal window displays the docker run command.](media//local_backend_docker_run.png "Local backend api Docker run") - -6. Open a browser and navigate to `http://localhost:4242/docs` to view the Swagger UI. - -7. Repeat steps 8-10 from the previous section to test the backend api running in a container on Docker Desktop. - -## Deploy the backend api to Azure Container Apps - -### Retrieve the Azure Container Registry login server and admin credentials - -The backend api container image needs to be pushed to an Azure Container Registry (ACR) before it can be deployed to Azure Container Apps. The ACR is a private container registry that will store the container image. Azure Container Apps will pull the container image from the ACR to deploy the backend api. - -1. In the Azure portal, open the provisioned resource group and locate and open the **Container Registry** resource. - -2. Select **Access keys** from the left-hand menu. Record the **Login server** value and the **Username** and **Password** values for later use. - - ![The Azure portal displays the Container Registry resource with the Access keys menu item selected. The login server, username and password values are highlighted.](media/acr_access_keys.png "Container Registry Access keys") - -### Push the backend api container image to the Azure Container Registry - -Earlier, the backend api container image was built locally. Now that the ACR login server and admin credentials are known, the container image can be pushed to the Azure Container Registry. - -1. Return to the terminal window in VS Code. - -2. Run the following command to tag the container image with the ACR login server. Replace the `` value. This command will silently complete with no output. - - ```bash - docker tag devguidenodebackendapi:latest /devguidenodebackendapi:v1 - ``` - -3. Run the following command to log into the ACR. Replace the ``, ``, and `` values. The message `Login Succeeded` displays when the login is successful. - - ```bash - docker login -u -p - ``` - -4. Once authenticated, push the container image to the ACR using the following command. Replace the `` value. - - ```bash - docker push /devguidenodebackendapi:v1 - ``` - - ![The VSCode terminal window displays the docker login and docker push commands.](media/local_backend_docker_push.png "Local backend api Docker push") - -### Deploy the backend api ACR container image to Azure Container Apps - -The last step is to deploy the backend api container image to Azure Container Apps. Azure Container Apps is a fully managed serverless container platform that allows developers to deploy containerized applications without having to manage any infrastructure. - -1. In the Azure portal, open the provisioned resource group and locate and open the **Container App** resource, the name will end in `-api`. Record the name of this resource. - -2. Back in the resource group, locate the **Container Apps Environment** resource, the name will end in `-containerappenv`. Record the name of this resource. - -3. Also record the name of the resource group. - -4. Return to the terminal window in VS Code. - -5. Log into the Azure CLI using the following command. - - ```bash - az login - ``` - -6. Optionally set the current subscription to the correct subscription using the following command. - - ```bash - az account set --subscription - ``` - -7. Install the Azure Container Apps extension for the CLI using the following command. - - ```bash - az extension add --name containerapp --upgrade - ``` - -8. Run the following command to deploy the backend api container image to the existing Azure Container Apps resource. Replace the ``, ``, ``, and `` values. - - ```bash - az containerapp up --name --image /devguidenodebackendapi:v1 --resource-group --environment --ingress external - ``` - - ![The console output of the container app up command displays with the endpoint highlighted.](media/container_deploy.png) - -9. In the Azure Portal, locate and open the **Container App** resource ending in `-api`. - -10. Notice on the **Overview** screen, there is a failed container revision. This is because the `hello-world` container is running at the same binding address as the backend api container. - - ![The Azure portal displays the Container App resource with a failed container revision listed.](media/container_app_failed_revision.png "Container App Failed Revision") - -11. View the error from the logs, or optionally select **Log stream** from the left menu, then select the api container log stream. - - ![The Azure portal displays the Container App resource with the Log stream menu item selected and the output of the api container with the error highlighted.](media/container_app_log_stream.png "Container App Log Stream") - -12. To rectify this, the `hello-world` container needs to be deleted. Select **Containers** from the left menu, then choose the **Edit and Deploy** button from the toolbar. - - ![The Azure portal displays the Container App resource with the Containers menu item selected. The Edit and Deploy button is highlighted.](media/container_app_edit_and_deploy.png "Container App Edit and Deploy") - -13. On the **Create and deploy new revision** screen, beneath the **Container image** heading, check the box next to the `hello-world` container, then select **Delete**. - - ![The Azure portal displays the Create and deploy new revision screen with the hello-world container selected.](media/container_app_delete_hello_world.png "Container App Delete hello-world") - -14. Select **Create** to deploy the new revision. In less than a minute, the new revision is deployed. - -15. Refresh the browser, then from the left menu, select **Overview**. Select the **Application URL** link. - - ![The Azure portal displays the Container App resource Overview screen with the Application URL highlighted.](media/container_app_overview.png "Container App Overview") - -16. The UI should show the status as `ready`. - - ![The Azure portal displays the Container App resource with the Overview menu item selected. The Application URL is highlighted.](media/container_app_ready.png "Container App Ready") - -17. In the address bar of the browser, append `/docs` to the URL and press ENTER to view the Swagger UI. - -18. Repeat steps 8-10 from the [Run the backend api locally section](#run-the-backend-api-locally) to test the backend api running in a container on Azure Container Apps. - -Congratulations on the successful deployment of the backend api to Azure Container Apps where it is ready to service the frontend application. \ No newline at end of file diff --git a/Labs/deploy/deploy.md b/Labs/deploy/deploy.md index 463f94c..8d53f64 100644 --- a/Labs/deploy/deploy.md +++ b/Labs/deploy/deploy.md @@ -9,7 +9,9 @@ ## Clone the repository -Create a folder to house the repository. Open a terminal and navigate to the folder. Clone the repository, then navigate to the `Labs/deploy` folder within the repository. +Create a folder to house the repository. +Open a terminal and navigate to the folder. +Clone the repository, then navigate to the `Labs/deploy` folder within the repository. ```bash git clone https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide.git diff --git a/Labs/explore_and_use_models/README.md b/Labs/lab_0_explore_and_use_models/README.md similarity index 90% rename from Labs/explore_and_use_models/README.md rename to Labs/lab_0_explore_and_use_models/README.md index 2dedde4..af971a8 100644 --- a/Labs/explore_and_use_models/README.md +++ b/Labs/lab_0_explore_and_use_models/README.md @@ -2,7 +2,8 @@ ## OpenAI Client Library -When integrating Azure OpenAI service in a solution written in Node.js, the OpenAI NPM client library is used. This library is maintained by OpenAI, and is compatible with the Azure OpenAI service. +When integrating Azure OpenAI service in a solution written in Node.js, the OpenAI NPM client library is used. +This library is maintained by OpenAI, and is compatible with the Azure OpenAI service. Install the latest openai client library using `npm`: @@ -40,7 +41,8 @@ const chatResponse = client.getChatCompletions( ]); ``` -The `.getCompletions` method returns a Javascript Promise object for asynchronous programming. In this example the next step is to call the `.then` method on the promise, passing it a function that writes the response from Azure OpenAI to the console. +The `.getCompletions` method returns a Javascript Promise object for asynchronous programming. +In this example the next step is to call the `.then` method on the promise, passing it a function that writes the response from Azure OpenAI to the console. ```javascript chatResponse.then((result) => { diff --git a/Labs/explore_and_use_models/completed/app.js b/Labs/lab_0_explore_and_use_models/completed/app.js similarity index 100% rename from Labs/explore_and_use_models/completed/app.js rename to Labs/lab_0_explore_and_use_models/completed/app.js diff --git a/Labs/explore_and_use_models/completed/package.json b/Labs/lab_0_explore_and_use_models/completed/package.json similarity index 100% rename from Labs/explore_and_use_models/completed/package.json rename to Labs/lab_0_explore_and_use_models/completed/package.json diff --git a/Labs/first_cosmos_db_application/.env.example b/Labs/lab_1_first_application/.env.example similarity index 100% rename from Labs/first_cosmos_db_application/.env.example rename to Labs/lab_1_first_application/.env.example diff --git a/Labs/first_cosmos_db_application/README.md b/Labs/lab_1_first_application/README.md similarity index 53% rename from Labs/first_cosmos_db_application/README.md rename to Labs/lab_1_first_application/README.md index 8258019..393305c 100644 --- a/Labs/first_cosmos_db_application/README.md +++ b/Labs/lab_1_first_application/README.md @@ -1,36 +1,45 @@ # Lab - Create your first Cosmos DB for MongoDB application -This lab demonstrates the creation of a Cosmos DB for MongoDB application. The lab will cover creating a database client and using that client to create a database, collection, and documents. The lab will also cover querying for documents and updating and deleting documents. +This lab demonstrates the creation of a Cosmos DB for MongoDB application. +The lab will cover creating a database client and using that client to create a database, collection, and documents. +The lab will also cover querying for documents and updating and deleting documents. ## Create a MongoDB client and connect to the Cosmos DB for MongoDB API service -1. In the lab folder, create a `.env` file and add the following environment variables, replace `` with your Cosmos DB for MongoDB API service connection string: +1. +In the lab folder, create a `.env` file and add the following environment variables, replace `` with your Cosmos DB for MongoDB API service connection string: ```text MONGODB_URI= ``` -2. In Visual Studio Code, open a terminal window and navigate to the lab folder `first_cosmos_db_application`. +2. +In Visual Studio Code, open a terminal window and navigate to the lab folder `lab_1_first_application`. -3. Install the required packages by running the following command in the terminal window: +3. +Install the required packages by running the following command in the terminal window: ```bash npm install ``` -4. Open the `index.js` file, at the top of the file, add the following code to import the MongoDB client from the **mongodb** package: +4. +Open the `index.js` file, at the top of the file, add the following code to import the MongoDB client from the **mongodb** package: ```javascript const { MongoClient } = require('mongodb'); ``` -5. In the `main` function, add the following code to initialize the MongoDB client. +5. +In the `main` function, add the following code to initialize the MongoDB client. ```javascript const client = new MongoClient(process.env.MONGODB_URI); ``` -6. Directly beneath the client initialization, add the following code to connect to the Cosmos DB for MongoDB database service using the client. This code connects to the database service and outputs messages to the console to indicate the connection status. +6. +Directly beneath the client initialization, add the following code to connect to the Cosmos DB for MongoDB database service using the client. +This code connects to the database service and outputs messages to the console to indicate the connection status. ```javascript try { @@ -44,7 +53,8 @@ This lab demonstrates the creation of a Cosmos DB for MongoDB application. The l } ``` -7. Verify the code listing in the `index.js` file matches the following code: +7. +Verify the code listing in the `index.js` file matches the following code: ```javascript require('dotenv').config(); @@ -65,7 +75,9 @@ This lab demonstrates the creation of a Cosmos DB for MongoDB application. The l main().catch(console.error); ``` -8. Save the changes to the file, and run the application by running the following command in the terminal window. This command will output the connection success and disconnection messages to the console. +8. +Save the changes to the file, and run the application by running the following command in the terminal window. +This command will output the connection success and disconnection messages to the console. ```bash npm start @@ -75,33 +87,49 @@ This lab demonstrates the creation of a Cosmos DB for MongoDB application. The l ## Create a MongoDB database -When using the `mongodb` client, the creation of a database is automatic when referenced. No specific api calls to create a database are required, if a database already exists, a reference to the database is returned. +When using the `mongodb` client, the creation of a database is automatic when referenced. +No specific api calls to create a database are required, if a database already exists, a reference to the database is returned. -1. In the `index.js` file, after the database connection code, add the following code to obtain a reference to a database. Save the file. +1. +In the `index.js` file, after the database connection code, add the following code to obtain a reference to a database. +Save the file. ```javascript const db = client.db('cosmic_works'); ``` ->**Note:**: That the creation of databases and collections are lazy, meaning they will not be created until a document is inserted into a collection. Running the application now will not create the database. +>**Note:**: That the creation of databases and collections are lazy, meaning they will not be created until a document is inserted into a collection. +Running the application now will not create the database. ## Create a collection -Creating collections behaves similarly to the database creation. If the collection does not exist, it will be created. It's important to note that databases and collections are lazily created. This means that the database and collection will not be created until the first document is inserted. +Creating collections behaves similarly to the database creation. +If the collection does not exist, it will be created. +It's important to note that databases and collections are lazily created. +This means that the database and collection will not be created until the first document is inserted. -1. In the `index.js` file, after the database reference code, add the following code to obtain a reference to a collection. Save the file. +1. +In the `index.js` file, after the database reference code, add the following code to obtain a reference to a collection. +Save the file. ```javascript const products = db.collection('products'); ``` ->**Note:**: Remember that the creation of databases and collections are lazy, meaning they will not be created until a document is inserted into a collection. Running the application now will not create the database or the collection at this point. +>**Note:**: Remember that the creation of databases and collections are lazy, meaning they will not be created until a document is inserted into a collection. +Running the application now will not create the database or the collection at this point. ## Create a document -Documents in Cosmos DB API for MongoDB are represented as JSON objects. One method of creating a document is using the `insertOne` method. This method takes a single document and inserts it into the database. This operation returns an [InsertOneResult](https://mongodb.github.io/node-mongodb-native/6.3/interfaces/InsertOneResult.html) object that contains the property **insertedId**. This property contains the unique identifier of the document that was just inserted. +Documents in Cosmos DB API for MongoDB are represented as JSON objects. +One method of creating a document is using the `insertOne` method. +This method takes a single document and inserts it into the database. +This operation returns an [InsertOneResult](https://mongodb.github.io/node-mongodb-native/6.3/interfaces/InsertOneResult.html) object that contains the property **insertedId**. +This property contains the unique identifier of the document that was just inserted. -1. In the `index.js` file, after the collection reference code, add the following code to insert a document into the collection. Save the file. +1. +In the `index.js` file, after the collection reference code, add the following code to insert a document into the collection. +Save the file. ```javascript const product = { @@ -117,7 +145,9 @@ Documents in Cosmos DB API for MongoDB are represented as JSON objects. One meth console.log(`A product was inserted with the _id: ${result.insertedId}`); ``` -2. Run the application. In addition to the connectivity messages in addition to the document insertion message. +2. +Run the application. +In addition to the connectivity messages in addition to the document insertion message. ```bash npm start @@ -125,20 +155,30 @@ Documents in Cosmos DB API for MongoDB are represented as JSON objects. One meth ![Console output showing a successful connection, document insertion, and disconnection messages.](media/db_insert_output.png "Database document insertion console output") -3. Return to `index.js` and delete or comment out the code from step 1 of this section. This will ensure that the same record gets re-added each time the application is run. +3. +Return to `index.js` and delete or comment out the code from step 1 of this section. +This will ensure that the same record gets re-added each time the application is run. ## Read a document -The insertion of the product in the previous section automatically created the database and collection. The `findOne` method is used to retrieve a single document from the database. The `findOne` method takes a filter as an argument. This filter is used to find the document in the database. In this case, the filter is the unique identifier or **_id** of the document that was just inserted. +The insertion of the product in the previous section automatically created the database and collection. +The `findOne` method is used to retrieve a single document from the database. +The `findOne` method takes a filter as an argument. +This filter is used to find the document in the database. +In this case, the filter is the unique identifier or **_id** of the document that was just inserted. -1. In the `index.js` file, after the collection reference code, add the following code to retrieve a document from the collection. Save the file. +1. +In the `index.js` file, after the collection reference code, add the following code to retrieve a document from the collection. +Save the file. ```javascript const product = await products.findOne({ _id: '2BA4A26C-A8DB-4645-BEB9-F7D42F50262E' }); console.log("Found the product:", product); ``` -2. Run the application. In addition to the connectivity messages, the document retrieval message will be displayed. +2. +Run the application. +In addition to the connectivity messages, the document retrieval message will be displayed. ```bash npm start @@ -148,9 +188,16 @@ The insertion of the product in the previous section automatically created the d ## Update a document -The `findOneAndUpdate` method is used to update a single document in the database. This method takes a filter and an update as arguments. The filter is used to find the document to update. The update is a dictionary of the properties to update. In this case, the `findOneAndUpdate` method is used to update the name property of the document. The updated document is the return value for this method. +The `findOneAndUpdate` method is used to update a single document in the database. +This method takes a filter and an update as arguments. +The filter is used to find the document to update. +The update is a dictionary of the properties to update. +In this case, the `findOneAndUpdate` method is used to update the name property of the document. +The updated document is the return value for this method. -1. In the `index.js` file, after the collection reference code and before the `findOne` code, add the following code to update a document in the collection. Save the file. +1. +In the `index.js` file, after the collection reference code and before the `findOne` code, add the following code to update a document in the collection. +Save the file. ```javascript const options = { returnDocument: 'after' }; @@ -160,7 +207,9 @@ The `findOneAndUpdate` method is used to update a single document in the databas options); ``` -2. Run the application. In addition to the connectivity messages, the document retrieved message will be displayed with the updated price. +2. +Run the application. +In addition to the connectivity messages, the document retrieved message will be displayed with the updated price. ```bash npm start @@ -168,20 +217,30 @@ The `findOneAndUpdate` method is used to update a single document in the databas ![Console output displays the connectivity messages as well as the retrieval of the document message with the updated price.](media/db_update_output.png "Document is updated.") -3. Remove the code that was just added in this section from the `index.js` file and save the file. +3. +Remove the code that was just added in this section from the `index.js` file and save the file. ## Delete a document -The `deleteOne` method is used to delete a single document from the database. This method takes a filter as an argument. This filter is used to find the document to delete. In this case, the filter is the unique identifier or `_id` of the document that was just inserted and updated. The `deleteOne` method returns a [`DeleteResult`](https://mongodb.github.io/node-mongodb-native/6.3/interfaces/DeleteResult.html) object. +The `deleteOne` method is used to delete a single document from the database. +This method takes a filter as an argument. +This filter is used to find the document to delete. +In this case, the filter is the unique identifier or `_id` of the document that was just inserted and updated. +The `deleteOne` method returns a [`DeleteResult`](https://mongodb.github.io/node-mongodb-native/6.3/interfaces/DeleteResult.html) object. -1. In the `index.js` file, after the collection reference code and before the `findOne` code, add the following code to delete a document from the collection. Save the file. +1. +In the `index.js` file, after the collection reference code and before the `findOne` code, add the following code to delete a document from the collection. +Save the file. ```javascript const result = await products.deleteOne({ _id: '2BA4A26C-A8DB-4645-BEB9-F7D42F50262E' }); console.log(`Number of documents deleted: ${result.deletedCount}`); ``` -2. Run the application. In addition to the connectivity messages, the document deletion message will be displayed. Note the retrieval message displays `null` as the document was deleted. +2. +Run the application. +In addition to the connectivity messages, the document deletion message will be displayed. +Note the retrieval message displays `null` as the document was deleted. ```bash npm start @@ -189,13 +248,22 @@ The `deleteOne` method is used to delete a single document from the database. Th ![Console output displays the connectivity messages as well as the deletion of the document message.](media/db_delete_output.png "Document is deleted.") -3. Remove the code that was just added in this section, as well as the `findOne` code from the `index.js` file and save the file. +3. +Remove the code that was just added in this section, as well as the `findOne` code from the `index.js` file and save the file. ## Query for multiple documents -In this section, multiple product documents are written to the database using a `bulkWrite` operation. Bulk operations will be covered in more detail in the next lab. Having multiple documents in the products collection will allow for querying for multiple documents using the `find` method. This method takes a filter as an argument. This filter is used to find the documents to return. In this case, the filter is empty and therefore will return all documents in the collection. The `find` method returns a cursor that can be converted to an array of documents using the `toArray` method. +In this section, multiple product documents are written to the database using a `bulkWrite` operation. +Bulk operations will be covered in more detail in the next lab. +Having multiple documents in the products collection will allow for querying for multiple documents using the `find` method. +This method takes a filter as an argument. +This filter is used to find the documents to return. +In this case, the filter is empty and therefore will return all documents in the collection. +The `find` method returns a cursor that can be converted to an array of documents using the `toArray` method. -1. In the `index.js` file, add the following code to insert multiple product documents in the database. Place this code directly beneath the collection reference code. +1. +In the `index.js` file, add the following code to insert multiple product documents in the database. +Place this code directly beneath the collection reference code. ```javascript const productsToInsert = [ @@ -255,14 +323,18 @@ In this section, multiple product documents are written to the database using a console.log("Bulk write operation result:", result); ``` -2. Immediately following the preceding code, add the following code to query for multiple documents in the collection. Save the file. +2. +Immediately following the preceding code, add the following code to query for multiple documents in the collection. +Save the file. ```javascript const allProducts = await products.find({}).toArray(); console.log("Found the following products:", allProducts); ``` -3. Run the application. In addition to the connectivity messages, the bulk write operation result showing the inserted id values and the retrieval of all documents output to the console. +3. +Run the application. +In addition to the connectivity messages, the bulk write operation result showing the inserted id values and the retrieval of all documents output to the console. ```bash npm start @@ -272,19 +344,24 @@ In this section, multiple product documents are written to the database using a ![The second part of the console output displays the retrieval of all documents.](media/db_retrieve_all_output.png "Retrieval of all documents") -4. Remove all code added in this section, and save the `index.js` file. +4. +Remove all code added in this section, and save the `index.js` file. ## Clean up resources Clean up the items created in this lab by deleting the MongoDB database. -1. In `index.js`, add the following code after the collection reference code to delete the database. Save the file. +1. +In `index.js`, add the following code after the collection reference code to delete the database. +Save the file. ```javascript await db.dropDatabase(); ``` -2. Run the application. In addition to the connectivity messages, the database deletion message will be displayed. +2. +Run the application. +In addition to the connectivity messages, the database deletion message will be displayed. ```bash npm start diff --git a/Labs/first_cosmos_db_application/index.js b/Labs/lab_1_first_application/index.js similarity index 100% rename from Labs/first_cosmos_db_application/index.js rename to Labs/lab_1_first_application/index.js diff --git a/Labs/first_cosmos_db_application/media/db_bulk_write_output.png b/Labs/lab_1_first_application/media/db_bulk_write_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_bulk_write_output.png rename to Labs/lab_1_first_application/media/db_bulk_write_output.png diff --git a/Labs/first_cosmos_db_application/media/db_connection_output.png b/Labs/lab_1_first_application/media/db_connection_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_connection_output.png rename to Labs/lab_1_first_application/media/db_connection_output.png diff --git a/Labs/first_cosmos_db_application/media/db_delete_output.png b/Labs/lab_1_first_application/media/db_delete_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_delete_output.png rename to Labs/lab_1_first_application/media/db_delete_output.png diff --git a/Labs/first_cosmos_db_application/media/db_insert_output.png b/Labs/lab_1_first_application/media/db_insert_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_insert_output.png rename to Labs/lab_1_first_application/media/db_insert_output.png diff --git a/Labs/first_cosmos_db_application/media/db_retrieve_all_output.png b/Labs/lab_1_first_application/media/db_retrieve_all_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_retrieve_all_output.png rename to Labs/lab_1_first_application/media/db_retrieve_all_output.png diff --git a/Labs/first_cosmos_db_application/media/db_retrieve_output.png b/Labs/lab_1_first_application/media/db_retrieve_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_retrieve_output.png rename to Labs/lab_1_first_application/media/db_retrieve_output.png diff --git a/Labs/first_cosmos_db_application/media/db_update_output.png b/Labs/lab_1_first_application/media/db_update_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/db_update_output.png rename to Labs/lab_1_first_application/media/db_update_output.png diff --git a/Labs/first_cosmos_db_application/media/delete_database_output.png b/Labs/lab_1_first_application/media/delete_database_output.png similarity index 100% rename from Labs/first_cosmos_db_application/media/delete_database_output.png rename to Labs/lab_1_first_application/media/delete_database_output.png diff --git a/Labs/first_cosmos_db_application/package.json b/Labs/lab_1_first_application/package.json similarity index 100% rename from Labs/first_cosmos_db_application/package.json rename to Labs/lab_1_first_application/package.json diff --git a/Labs/load_data/.env.example b/Labs/lab_2_load_data/.env.example similarity index 100% rename from Labs/load_data/.env.example rename to Labs/lab_2_load_data/.env.example diff --git a/Labs/load_data/README.md b/Labs/lab_2_load_data/README.md similarity index 60% rename from Labs/load_data/README.md rename to Labs/lab_2_load_data/README.md index b11687c..9eeb34e 100644 --- a/Labs/load_data/README.md +++ b/Labs/lab_2_load_data/README.md @@ -4,15 +4,18 @@ This lab demonstrates bulk loading of data from the Cosmic Works JSON files into ## Setup the lab environment -1. In the lab folder, create a `.env` file and add the following environment variables, replace `` with your Cosmos DB for MongoDB API service connection string: +1. +In the lab folder, create a `.env` file and add the following environment variables, replace `` with your Cosmos DB for MongoDB API service connection string: ```text MONGODB_URI= ``` -2. In Visual Studio Code, open a terminal window and navigate to the lab folder `load_data`. +2. +In Visual Studio Code, open a terminal window and navigate to the lab folder `lab_2_load_data`. -3. Install the required packages by running the following command in the terminal window: +3. +Install the required packages by running the following command in the terminal window: ```bash npm install @@ -20,9 +23,12 @@ This lab demonstrates bulk loading of data from the Cosmic Works JSON files into ## Bulk load product data -There is more than one option when performing bulk operations in Cosmos DB for MongoDB. In this section, data will be loaded using the `bulkWrite` method. The `bulkWrite` method is used to perform multiple write operations in a single batch, write operations can include a mixture of insert, update, and delete operations. +There is more than one option when performing bulk operations in Cosmos DB for MongoDB. +In this section, data will be loaded using the `bulkWrite` method. +The `bulkWrite` method is used to perform multiple write operations in a single batch, write operations can include a mixture of insert, update, and delete operations. -1. Open the `index.js` file, and directly beneath the `const db = client.db('cosmic_works');` line, add the following code to fetch the product data from the Cosmic Works repository: +1. +Open the `index.js` file, and directly beneath the `const db = client.db('cosmic_works');` line, add the following code to fetch the product data from the Cosmic Works repository: ```javascript // Load product data @@ -35,7 +41,9 @@ There is more than one option when performing bulk operations in Cosmos DB for M .map(prod => cleanData(prod)); ``` -2. Optionally, append the following code (to the code in the previous step) to delete any existing products in the collection. This helps if the application is run multiple times so there is no duplicates. +2. +Optionally, append the following code (to the code in the previous step) to delete any existing products in the collection. +This helps if the application is run multiple times so there is no duplicates. ```javascript // Delete any existing products @@ -43,7 +51,8 @@ There is more than one option when performing bulk operations in Cosmos DB for M await productCollection.deleteMany({}); ``` -3. Append the following code (to the code in the previous step) to bulk load the product data into the collection: +3. +Append the following code (to the code in the previous step) to bulk load the product data into the collection: ```javascript var result = await productCollection.bulkWrite( @@ -56,9 +65,11 @@ There is more than one option when performing bulk operations in Cosmos DB for M console.log(`${result.insertedCount} products inserted`); ``` -4. Save the `index.js` file. +4. +Save the `index.js` file. -5. Run the application by executing the following command in the terminal window: +5. +Run the application by executing the following command in the terminal window: ```bash npm start @@ -68,11 +79,13 @@ There is more than one option when performing bulk operations in Cosmos DB for M ## Bulk load of customer and sales data -In this section, data will be loaded using the `insertMany` method. The `insertMany` method is used to insert multiple documents into a collection, it differs from the `bulkWrite` method in that it only supports insert operations. +In this section, data will be loaded using the `insertMany` method. +The `insertMany` method is used to insert multiple documents into a collection, it differs from the `bulkWrite` method in that it only supports insert operations. Customer data and sales data are also combined in a single JSON source, some pre-processing is required to separate the data into two separate collections. -1. Open the `index.js` file, and directly beneath the code for adding products, append the following code to fetch the customer and sales data from the Cosmic Works repository: +1. +Open the `index.js` file, and directly beneath the code for adding products, append the following code to fetch the customer and sales data from the Cosmic Works repository: ```javascript // Load customer and sales data @@ -84,7 +97,8 @@ Customer data and sales data are also combined in a single JSON source, some pre .map(custSales => cleanData(custSales)); ``` -2. Split the customer data from the sales data by appending the following code (to the code in the previous step): +2. +Split the customer data from the sales data by appending the following code (to the code in the previous step): ```javascript console.log("Split customer and sales data"); @@ -92,7 +106,8 @@ Customer data and sales data are also combined in a single JSON source, some pre const salesData = custSalesData.filter(sales => sales["type"] === "salesOrder"); ``` -3. Append the following code (to the code in the previous step) to bulk load the customer data into the collection using the `insertMany` method: +3. +Append the following code (to the code in the previous step) to bulk load the customer data into the collection using the `insertMany` method: ```javascript console.log("Loading customer data"); @@ -101,7 +116,8 @@ Customer data and sales data are also combined in a single JSON source, some pre console.log(`${result.insertedCount} customers inserted`); ``` -4. Append the following code (to the code in the previous step) to bulk load the sales data into the collection using the `insertMany` method: +4. +Append the following code (to the code in the previous step) to bulk load the sales data into the collection using the `insertMany` method: ```javascript console.log("Loading sales data"); @@ -110,9 +126,11 @@ Customer data and sales data are also combined in a single JSON source, some pre console.log(`${result.insertedCount} sales inserted`); ``` -5. Save the `index.js` file. +5. +Save the `index.js` file. -6. Run the application by executing the following command in the terminal window: +6. +Run the application by executing the following command in the terminal window: ```bash npm start @@ -122,4 +140,5 @@ Customer data and sales data are also combined in a single JSON source, some pre ## Summary -In this section bulk load operations were used to load product, customer, and sales data into Cosmos DB for MongoDB. Keep the database and its loaded data for use in subsequent labs. +In this section bulk load operations were used to load product, customer, and sales data into Cosmos DB for MongoDB. +Keep the database and its loaded data for use in subsequent labs. diff --git a/Labs/lab_2_load_data/index.js b/Labs/lab_2_load_data/index.js new file mode 100644 index 0000000..e7c1175 --- /dev/null +++ b/Labs/lab_2_load_data/index.js @@ -0,0 +1,55 @@ +require("dotenv").config(); +const { MongoClient } = require("mongodb"); +const fs = require("fs"); + +async function main() { + const client = new MongoClient(process.env.MONGODB_URI); + try { + await client.connect(); + console.log("Connected to MongoDB"); + const db = client.db("cosmic_works"); + console.log("Using database cosmic_works"); + + // Load product data + console.log("Loading product data"); + // Initialize the product collection pointer (will automatically be created if it doesn't exist) + const productCollection = db.collection("products"); + // Load the product data from the local product.json file while also removing the system properties + const productRawData = fs.readFileSync("product.json", "utf8"); + const productData = JSON.parse(productRawData).map((prod) => + cleanData(prod) + ); + + // // Delete any existing products + // console.log("Deleting existing products"); + // await productCollection.deleteMany({}); + + var result = await productCollection.bulkWrite( + productData.map((product) => ({ + updateOne: { + filter: { sku: product.sku }, + update: { $set: product }, + upsert: true, + }, + })) + ); + console.log(`${result.upsertedCount} products upserted`); + } catch (err) { + console.error(err); + } finally { + await client.close(); + console.log("Disconnected from MongoDB"); + } +} + +function cleanData(obj) { + cleaned = Object.fromEntries( + Object.entries(obj).filter(([key, _]) => !key.startsWith("_")) + ); + //rename id field to _id + cleaned["_id"] = cleaned["id"]; + delete cleaned["id"]; + return cleaned; +} + +main().catch(console.error); diff --git a/Labs/load_data/media/customers_sales_loaded.png b/Labs/lab_2_load_data/media/customers_sales_loaded.png similarity index 100% rename from Labs/load_data/media/customers_sales_loaded.png rename to Labs/lab_2_load_data/media/customers_sales_loaded.png diff --git a/Labs/load_data/media/products_loaded.png b/Labs/lab_2_load_data/media/products_loaded.png similarity index 100% rename from Labs/load_data/media/products_loaded.png rename to Labs/lab_2_load_data/media/products_loaded.png diff --git a/Labs/load_data/package.json b/Labs/lab_2_load_data/package.json similarity index 100% rename from Labs/load_data/package.json rename to Labs/lab_2_load_data/package.json diff --git a/Labs/lab_2_load_data/product.json b/Labs/lab_2_load_data/product.json new file mode 100644 index 0000000..7e4d9fa --- /dev/null +++ b/Labs/lab_2_load_data/product.json @@ -0,0 +1,2362 @@ +[ + { + "id": "063F1A00-8CA1-4DB9-8298-BEAC4B8CC238", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R79Y-48", + "name": "Road-350-W Yellow, 48", + "description": "The product called \"Road-350-W Yellow, 48\"", + "price": 1700.99 + }, + { + "id": "06AC4FFF-9F97-429B-BB15-ED929EFF65EE", + "categoryId": "C48B4EF4-D352-4CD2-BCB8-CE89B7DFA642", + "sku": "SO-R809-M", + "name": "Racing Socks, M", + "description": "The product called \"Racing Socks, M\"", + "price": 8.99 + }, + { + "id": "0846D2C3-7E50-4F68-A6CB-F0DC90FD03D0", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-T905", + "name": "Touring Front Wheel", + "description": "The product called \"Touring Front Wheel\"", + "price": 218.01 + }, + { + "id": "0B013EA7-B40E-4996-A494-D1E2840FEAAE", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72R-44", + "name": "ML Road Frame - Red, 44", + "description": "The product called \"ML Road Frame - Red, 44\"", + "price": 594.83 + }, + { + "id": "0C3D95EB-EE37-44A5-816F-957A98519B03", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50B-44", + "name": "Road-650 Black, 44", + "description": "The product called \"Road-650 Black, 44\"", + "price": 782.99 + }, + { + "id": "0D7CB85D-4518-4E02-8E46-9683947BBBC4", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R68R-52", + "name": "Road-450 Red, 52", + "description": "The product called \"Road-450 Red, 52\"", + "price": 1457.99 + }, + { + "id": "0E92DDAC-F969-4F63-8D5E-614AB5199D01", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89R-44", + "name": "Road-250 Red, 44", + "description": "The product called \"Road-250 Red, 44\"", + "price": 2443.35 + }, + { + "id": "11E6FD95-0FF1-4FE8-9A6B-EC53F614212D", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R19B-58", + "name": "Road-750 Black, 58", + "description": "The product called \"Road-750 Black, 58\"", + "price": 539.99 + }, + { + "id": "18711AD6-0999-4E74-B2F5-81720A6BA5A2", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-T928", + "name": "HL Touring Handlebars", + "description": "The product called \"HL Touring Handlebars\"", + "price": 91.57 + }, + { + "id": "18B722BF-4742-4F1F-8336-3AB2E76B2908", + "categoryId": "E048A761-8038-42C2-8367-F21FF0DAA3F4", + "sku": "FE-6654", + "name": "Fender Set - Mountain", + "description": "The product called \"Fender Set - Mountain\"", + "price": 21.98 + }, + { + "id": "201D0D79-81AD-43D2-AD6E-F09EEE6AC2D7", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-M798", + "name": "ML Mountain Seat/Saddle", + "description": "The product called \"ML Mountain Seat/Saddle\"", + "price": 39.14 + }, + { + "id": "209B4171-CB26-4231-8F41-D092F4679BB9", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94S-52", + "name": "HL Mountain Frame - Silver, 48", + "description": "The product called \"HL Mountain Frame - Silver, 48\"", + "price": 1364.5 + }, + { + "id": "24BE4267-85D8-4C1A-B184-C08709495752", + "categoryId": "006A1D51-28DA-4956-A7FB-C0B2BF6360CA", + "sku": "BC-R205", + "name": "Road Bottle Cage", + "description": "The product called \"Road Bottle Cage\"", + "price": 8.99 + }, + { + "id": "2C981511-AC73-4A65-9DA3-A0577E386394", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79U-46", + "name": "Touring-1000 Blue, 46", + "description": "The product called \"Touring-1000 Blue, 46\"", + "price": 2384.07 + }, + { + "id": "2FBE9F71-86EC-4FEB-BBF4-5580FD28E3FD", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-M897-X", + "name": "Men's Sports Shorts, XL", + "description": "The product called \"Men's Sports Shorts, XL\"", + "price": 59.99 + }, + { + "id": "32B61AF2-53BE-4E36-85D8-A24738769352", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18S-52", + "name": "Mountain-500 Silver, 52", + "description": "The product called \"Mountain-500 Silver, 52\"", + "price": 564.99 + }, + { + "id": "3933505E-7BD5-458D-84FE-546AA3520A66", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R68R-44", + "name": "Road-450 Red, 44", + "description": "The product called \"Road-450 Red, 44\"", + "price": 1457.99 + }, + { + "id": "397635D8-D71F-47B2-AD68-4ECA6A03F84F", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18B-44", + "name": "Mountain-500 Black, 44", + "description": "The product called \"Mountain-500 Black, 44\"", + "price": 539.99 + }, + { + "id": "409BC0E0-2B43-4F82-9C36-2E4ABBB7344C", + "categoryId": "ECEEC6AC-3CF1-41A6-8430-A1255F355BB5", + "sku": "FB-9873", + "name": "Front Brakes", + "description": "The product called \"Front Brakes\"", + "price": 106.5 + }, + { + "id": "462F8EAF-0988-4D32-B809-EB4362AF48D0", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M68B-38", + "name": "Mountain-200 Black, 38", + "description": "The product called \"Mountain-200 Black, 38\"", + "price": 2294.99 + }, + { + "id": "47ED1C3B-C205-4507-94EE-3B69A744B261", + "categoryId": "14A1AD5D-59EA-4B63-A189-67B077783B0E", + "sku": "HL-U509", + "name": "Sport-100 Helmet, Black", + "description": "The product called \"Sport-100 Helmet, Black\"", + "price": 34.99 + }, + { + "id": "491834BE-AAA5-419D-B166-77B93F20EBA7", + "categoryId": "AA5A82D4-914C-4132-8C08-E7B75DCE3428", + "sku": "CS-9183", + "name": "HL Crankset", + "description": "The product called \"HL Crankset\"", + "price": 404.99 + }, + { + "id": "49ACE2DB-4315-4C16-819E-BE372922C634", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-R820", + "name": "HL Road Rear Wheel", + "description": "The product called \"HL Road Rear Wheel\"", + "price": 357.06 + }, + { + "id": "4E4B38CB-0D82-43E5-89AF-20270CD28A04", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T44U-60", + "name": "Touring-2000 Blue, 60", + "description": "The product called \"Touring-2000 Blue, 60\"", + "price": 1214.85 + }, + { + "id": "5089E32E-8A60-4117-AA98-5EF8AB9A61D1", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TT-R982", + "name": "Road Tire Tube", + "description": "The product called \"Road Tire Tube\"", + "price": 3.99 + }, + { + "id": "5253671B-E50E-4686-9A17-4F51C2B65C0F", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72Y-40", + "name": "ML Road Frame-W - Yellow, 40", + "description": "The product called \"ML Road Frame-W - Yellow, 40\"", + "price": 594.83 + }, + { + "id": "55594B1E-1E16-4B2E-A16F-983E492321BC", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67Y-54", + "name": "LL Touring Frame - Yellow, 54", + "description": "The product called \"LL Touring Frame - Yellow, 54\"", + "price": 333.42 + }, + { + "id": "56BB7DD2-2421-4671-A527-7373008DD553", + "categoryId": "975E2A45-DA17-45CE-B65E-575A19334EB2", + "sku": "RD-2308", + "name": "Rear Derailleur", + "description": "The product called \"Rear Derailleur\"", + "price": 121.46 + }, + { + "id": "58978B2E-D4C6-4D69-A840-D935688F9C2D", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92R-56", + "name": "HL Road Frame - Red, 56", + "description": "The product called \"HL Road Frame - Red, 56\"", + "price": 1431.5 + }, + { + "id": "61B55CE9-DEB4-49B3-AB55-0AAC11EBBBBF", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38R-58", + "name": "LL Road Frame - Red, 58", + "description": "The product called \"LL Road Frame - Red, 58\"", + "price": 337.22 + }, + { + "id": "6B41F665-5810-4AFD-8323-6106A8593EFC", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-M928", + "name": "HL Mountain Rear Wheel", + "description": "The product called \"HL Mountain Rear Wheel\"", + "price": 327.215 + }, + { + "id": "6EAA3D6B-A290-48C4-B3ED-D668261512CD", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63S-46", + "name": "ML Mountain Frame-W - Silver, 46", + "description": "The product called \"ML Mountain Frame-W - Silver, 46\"", + "price": 364.09 + }, + { + "id": "6F733A5D-9B66-4718-B69C-627DE4E164BA", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67Y-44", + "name": "LL Touring Frame - Yellow, 44", + "description": "The product called \"LL Touring Frame - Yellow, 44\"", + "price": 333.42 + }, + { + "id": "7133D6F6-C8FD-4AD1-83E8-5622D1746E25", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72R-58", + "name": "ML Road Frame - Red, 58", + "description": "The product called \"ML Road Frame - Red, 58\"", + "price": 594.83 + }, + { + "id": "71BC9DC2-A409-4B4A-A34B-FCBF1E596FCF", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R93R-48", + "name": "Road-150 Red, 48", + "description": "The product called \"Road-150 Red, 48\"", + "price": 3578.27 + }, + { + "id": "80D3630F-B661-4FD6-A296-CD03BB7A4A0C", + "categoryId": "629A8F3C-CFB0-4347-8DCC-505A4789876B", + "sku": "VE-C304-L", + "name": "Classic Vest, L", + "description": "The product called \"Classic Vest, L\"", + "price": 63.5 + }, + { + "id": "866F8033-A439-42D9-99EE-178C1285F13E", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18Y-44", + "name": "Touring-3000 Yellow, 44", + "description": "The product called \"Touring-3000 Yellow, 44\"", + "price": 742.35 + }, + { + "id": "8B363B8B-378E-402A-9E68-A935302000B8", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98Y-46", + "name": "HL Touring Frame - Yellow, 46", + "description": "The product called \"HL Touring Frame - Yellow, 46\"", + "price": 1003.91 + }, + { + "id": "8B541087-A7F5-43B1-AC9F-EEFB4F4ADAFA", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18B-42", + "name": "Mountain-500 Black, 42", + "description": "The product called \"Mountain-500 Black, 42\"", + "price": 539.99 + }, + { + "id": "8B8184BF-B79B-4157-BAA6-D30413BCC7A9", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-M762", + "name": "ML Mountain Rear Wheel", + "description": "The product called \"ML Mountain Rear Wheel\"", + "price": 236.025 + }, + { + "id": "916ACEDC-DCF4-4118-90C6-B9572D30714E", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72R-60", + "name": "ML Road Frame - Red, 60", + "description": "The product called \"ML Road Frame - Red, 60\"", + "price": 594.83 + }, + { + "id": "935EB2B7-8D50-4E20-B01A-570DBA674AD4", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82S-38", + "name": "Mountain-100 Silver, 38", + "description": "The product called \"Mountain-100 Silver, 38\"", + "price": 3399.99 + }, + { + "id": "9FCA9658-8506-4268-8539-DBAA65C51F41", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-T723", + "name": "Touring Tire", + "description": "The product called \"Touring Tire\"", + "price": 28.99 + }, + { + "id": "A13C5B23-34DF-41C7-849C-0BA623BEFE02", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79Y-50", + "name": "Touring-1000 Yellow, 50", + "description": "The product called \"Touring-1000 Yellow, 50\"", + "price": 2384.07 + }, + { + "id": "A9EFB9E2-8859-4401-B8A6-F7E2D5264FEE", + "categoryId": "AA28AE74-D57C-4B23-B5F7-F919E1C5844E", + "sku": "TG-W091-M", + "name": "Women's Tights, M", + "description": "The product called \"Women's Tights, M\"", + "price": 74.99 + }, + { + "id": "B0FE1D0A-CED1-49E8-9ACF-E289A631A4ED", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M47B-44", + "name": "Mountain-300 Black, 44", + "description": "The product called \"Mountain-300 Black, 44\"", + "price": 1079.99 + }, + { + "id": "B2AC17CB-A69E-462E-B72A-917CB544FF81", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-R628", + "name": "ML Road Tire", + "description": "The product called \"ML Road Tire\"", + "price": 24.99 + }, + { + "id": "B79B140D-4369-429B-8F20-E28F3ED7F82A", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98Y-54", + "name": "HL Touring Frame - Yellow, 54", + "description": "The product called \"HL Touring Frame - Yellow, 54\"", + "price": 1003.91 + }, + { + "id": "BD340F0A-F661-4ED8-B36F-FBA7623605D9", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18U-62", + "name": "Touring-3000 Blue, 62", + "description": "The product called \"Touring-3000 Blue, 62\"", + "price": 742.35 + }, + { + "id": "BF3E0E82-DCFC-4EA3-A71C-8C9EAA329E14", + "categoryId": "C48B4EF4-D352-4CD2-BCB8-CE89B7DFA642", + "sku": "SO-B909-M", + "name": "Mountain Bike Socks, M", + "description": "The product called \"Mountain Bike Socks, M\"", + "price": 9.5 + }, + { + "id": "C461038A-6DB6-4EC7-924F-ECA906259A6E", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R79Y-42", + "name": "Road-350-W Yellow, 42", + "description": "The product called \"Road-350-W Yellow, 42\"", + "price": 1700.99 + }, + { + "id": "CB038CA5-3728-4B59-B209-22FAB210F58B", + "categoryId": "4F2FD0D4-F0E5-4F9E-B049-861E6541B987", + "sku": "HY-1023-70", + "name": "Hydration Pack - 70 oz.", + "description": "The product called \"Hydration Pack - 70 oz.\"", + "price": 54.99 + }, + { + "id": "CB1F441C-90E4-4E0B-ABDA-E0D07AFC2E01", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89B-58", + "name": "Road-250 Black, 58", + "description": "The product called \"Road-250 Black, 58\"", + "price": 2443.35 + }, + { + "id": "CC8D2C8C-AB60-48BE-A019-33F633DB07CD", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "SJ-0194-X", + "name": "Short-Sleeve Classic Jersey, XL", + "description": "The product called \"Short-Sleeve Classic Jersey, XL\"", + "price": 53.99 + }, + { + "id": "CDFC37BB-8DB8-4D66-841D-7C3FF28B1F0A", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-T721", + "name": "LL Touring Handlebars", + "description": "The product called \"LL Touring Handlebars\"", + "price": 46.09 + }, + { + "id": "D1F006A3-C6C0-42A4-B479-FC3A510C9E9E", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63B-38", + "name": "ML Mountain Frame - Black, 38", + "description": "The product called \"ML Mountain Frame - Black, 38\"", + "price": 348.76 + }, + { + "id": "DE810086-817F-440C-9FEF-471083B8E4A0", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "LJ-0192-M", + "name": "Long-Sleeve Logo Jersey, M", + "description": "The product called \"Long-Sleeve Logo Jersey, M\"", + "price": 49.99 + }, + { + "id": "E9F21624-C055-4D5F-8C02-8F69C1EA0AEE", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50R-44", + "name": "Road-650 Red, 44", + "description": "The product called \"Road-650 Red, 44\"", + "price": 782.99 + }, + { + "id": "F1AA8B6D-4CF2-4DB2-BB17-997C2BD1A6AC", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R68R-58", + "name": "Road-450 Red, 58", + "description": "The product called \"Road-450 Red, 58\"", + "price": 1457.99 + }, + { + "id": "FB9A5084-F2B2-4C3B-9CF9-252873CABFF7", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M38S-46", + "name": "Mountain-400-W Silver, 46", + "description": "The product called \"Mountain-400-W Silver, 46\"", + "price": 769.49 + }, + { + "id": "FD00408C-57B1-431C-B1FA-2CAF41D87CD4", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R19B-44", + "name": "Road-750 Black, 44", + "description": "The product called \"Road-750 Black, 44\"", + "price": 539.99 + }, + { + "id": "FDEF01CB-5067-414F-B0A3-07FF8A4B80DD", + "categoryId": "14A1AD5D-59EA-4B63-A189-67B077783B0E", + "sku": "HL-U509-R", + "name": "Sport-100 Helmet, Red", + "description": "The product called \"Sport-100 Helmet, Red\"", + "price": 34.99 + }, + { + "id": "FEEFEE3B-6CB9-4A75-B896-5182531F661B", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R19B-52", + "name": "Road-750 Black, 52", + "description": "The product called \"Road-750 Black, 52\"", + "price": 539.99 + }, + { + "id": "08225A9E-F2B3-4FA3-AB08-8C70ADD6C3C2", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79U-50", + "name": "Touring-1000 Blue, 50", + "description": "The product called \"Touring-1000 Blue, 50\"", + "price": 2384.07 + }, + { + "id": "0A7E57DA-C73F-467F-954F-17B7AFD6227E", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-R563", + "name": "ML Road Pedal", + "description": "The product called \"ML Road Pedal\"", + "price": 62.09 + }, + { + "id": "12DD6F29-6AA2-4C03-8873-19581F97E9CD", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18B-48", + "name": "Mountain-500 Black, 48", + "description": "The product called \"Mountain-500 Black, 48\"", + "price": 539.99 + }, + { + "id": "14174164-F6C0-47FC-83FB-604C6A63408D", + "categoryId": "006A1D51-28DA-4956-A7FB-C0B2BF6360CA", + "sku": "BC-M005", + "name": "Mountain Bottle Cage", + "description": "The product called \"Mountain Bottle Cage\"", + "price": 9.99 + }, + { + "id": "16F9DF28-56B4-4185-9B82-B85666BFA3A6", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M38S-38", + "name": "Mountain-400-W Silver, 38", + "description": "The product called \"Mountain-400-W Silver, 38\"", + "price": 769.49 + }, + { + "id": "1A176FDB-D9A8-4888-BDD9-CE4F12E97AAE", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-R982", + "name": "HL Road Tire", + "description": "The product called \"HL Road Tire\"", + "price": 32.6 + }, + { + "id": "21756241-F313-4D34-9914-9B7DAC76F9D6", + "categoryId": "973B839C-BF5D-485D-9D17-863C59B262E3", + "sku": "FK-1639", + "name": "LL Fork", + "description": "The product called \"LL Fork\"", + "price": 148.22 + }, + { + "id": "243AE98C-D657-415C-9EF4-D8FA8F8770AA", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50R-48", + "name": "Road-650 Red, 48", + "description": "The product called \"Road-650 Red, 48\"", + "price": 782.99 + }, + { + "id": "26E8185C-782A-4B48-87FA-1E715E3825FB", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R64Y-48", + "name": "Road-550-W Yellow, 48", + "description": "The product called \"Road-550-W Yellow, 48\"", + "price": 1120.49 + }, + { + "id": "2BA4A26C-A8DB-4645-BEB9-F7D42F50262E", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98U-50", + "name": "HL Touring Frame - Blue, 50", + "description": "The product called \"HL Touring Frame - Blue, 50\"", + "price": 1003.91 + }, + { + "id": "312A464A-1830-4755-8FB2-2ED32DC7FDD6", + "categoryId": "9268EA12-29BA-404B-B514-E4737DB3BFCB", + "sku": "SB-M891-L", + "name": "Men's Bib-Shorts, L", + "description": "The product called \"Men's Bib-Shorts, L\"", + "price": 89.99 + }, + { + "id": "32C5F63D-CF84-457C-9063-0C758CCDACE7", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92R-44", + "name": "HL Road Frame - Red, 44", + "description": "The product called \"HL Road Frame - Red, 44\"", + "price": 1431.5 + }, + { + "id": "34C0090C-B299-433B-8D31-42EFCDC5874D", + "categoryId": "345E8DEC-774F-45F6-BE0C-18CDDB368FC8", + "sku": "PA-T100", + "name": "Touring-Panniers, Large", + "description": "The product called \"Touring-Panniers, Large\"", + "price": 125.0 + }, + { + "id": "39F4BE10-8C68-4E7E-A185-B05BCA543B9F", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50B-52", + "name": "Road-650 Black, 52", + "description": "The product called \"Road-650 Black, 52\"", + "price": 782.99 + }, + { + "id": "3A70EDD4-6C8C-44AA-A13D-49D0F6058699", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R64Y-40", + "name": "Road-550-W Yellow, 40", + "description": "The product called \"Road-550-W Yellow, 40\"", + "price": 1120.49 + }, + { + "id": "3ADF5B22-B5B2-43CD-9E07-36A187EB9473", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "LJ-0192-S", + "name": "Long-Sleeve Logo Jersey, S", + "description": "The product called \"Long-Sleeve Logo Jersey, S\"", + "price": 49.99 + }, + { + "id": "3D9B62A3-3CDF-45A2-B64C-8A9890818E2C", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21B-48", + "name": "LL Mountain Frame - Black, 48", + "description": "The product called \"LL Mountain Frame - Black, 48\"", + "price": 249.79 + }, + { + "id": "3E144819-7455-4362-A4BB-FAD007A90AEF", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94S-44", + "name": "HL Mountain Frame - Silver, 44", + "description": "The product called \"HL Mountain Frame - Silver, 44\"", + "price": 1364.5 + }, + { + "id": "3FE1A99E-DE14-4D11-B635-F5D39258A0B9", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-T924", + "name": "HL Touring Seat/Saddle", + "description": "The product called \"HL Touring Seat/Saddle\"", + "price": 52.64 + }, + { + "id": "42FDA4EC-96CA-4160-956A-3870549AF76E", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R64Y-42", + "name": "Road-550-W Yellow, 42", + "description": "The product called \"Road-550-W Yellow, 42\"", + "price": 1120.49 + }, + { + "id": "44873725-7B3B-4B28-804D-963D2D62E761", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79U-60", + "name": "Touring-1000 Blue, 60", + "description": "The product called \"Touring-1000 Blue, 60\"", + "price": 2384.07 + }, + { + "id": "47C70E1E-E500-41B3-8615-DCCB963D9E35", + "categoryId": "AA28AE74-D57C-4B23-B5F7-F919E1C5844E", + "sku": "TG-W091-S", + "name": "Women's Tights, S", + "description": "The product called \"Women's Tights, S\"", + "price": 74.99 + }, + { + "id": "4973E28A-A70A-45B9-8517-5D3B647E82C2", + "categoryId": "BDC73EF8-1745-4A45-8944-D2868A763819", + "sku": "RA-H123", + "name": "Hitch Rack - 4-Bike", + "description": "The product called \"Hitch Rack - 4-Bike\"", + "price": 120.0 + }, + { + "id": "52FAD88C-567E-469D-A35E-574EA3BF147F", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-M340", + "name": "ML Mountain Pedal", + "description": "The product called \"ML Mountain Pedal\"", + "price": 62.09 + }, + { + "id": "58C93A21-73D1-44D8-ACF1-3A9E1DB0CE0D", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R93R-52", + "name": "Road-150 Red, 52", + "description": "The product called \"Road-150 Red, 52\"", + "price": 3578.27 + }, + { + "id": "5B5E90B8-FEA2-4D6C-B728-EC586656FA6D", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79Y-60", + "name": "Touring-1000 Yellow, 60", + "description": "The product called \"Touring-1000 Yellow, 60\"", + "price": 2384.07 + }, + { + "id": "5C24E8CD-2BFF-460A-88D4-3A2926407346", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-M918", + "name": "HL Mountain Handlebars", + "description": "The product called \"HL Mountain Handlebars\"", + "price": 120.27 + }, + { + "id": "5D3F5A52-A8BB-448C-B8CF-39D2FA2BDF3C", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98U-54", + "name": "HL Touring Frame - Blue, 54", + "description": "The product called \"HL Touring Frame - Blue, 54\"", + "price": 1003.91 + }, + { + "id": "5ED1BF5F-6C1F-4EF8-B1A7-B8A8412C9F72", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M47B-38", + "name": "Mountain-300 Black, 38", + "description": "The product called \"Mountain-300 Black, 38\"", + "price": 1079.99 + }, + { + "id": "600DDD58-C9D0-4118-9A69-B7716ED3A303", + "categoryId": "340D259D-BFFE-4E2A-9C5E-8B1E473A0322", + "sku": "ST-1401", + "name": "All-Purpose Bike Stand", + "description": "The product called \"All-Purpose Bike Stand\"", + "price": 159.0 + }, + { + "id": "626D67C2-C316-49EB-8316-129BDFBFDE8A", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50R-60", + "name": "Road-650 Red, 60", + "description": "The product called \"Road-650 Red, 60\"", + "price": 782.99 + }, + { + "id": "6374995F-9A78-43CD-AE0D-5F6041078140", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38R-60", + "name": "LL Road Frame - Red, 60", + "description": "The product called \"LL Road Frame - Red, 60\"", + "price": 337.22 + }, + { + "id": "637D953B-42DB-4219-927F-51687E889A04", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R93R-56", + "name": "Road-150 Red, 56", + "description": "The product called \"Road-150 Red, 56\"", + "price": 3578.27 + }, + { + "id": "6964ECD2-6FC5-4D65-88BC-126BC2BE2CCB", + "categoryId": "C48B4EF4-D352-4CD2-BCB8-CE89B7DFA642", + "sku": "SO-B909-L", + "name": "Mountain Bike Socks, L", + "description": "The product called \"Mountain Bike Socks, L\"", + "price": 9.5 + }, + { + "id": "6AEDC59D-F3E3-4B4F-9290-7EFC225B7F42", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-M928", + "name": "HL Mountain Front Wheel", + "description": "The product called \"HL Mountain Front Wheel\"", + "price": 300.215 + }, + { + "id": "6E059A32-56B5-4D98-AC6A-945B488B32A1", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R93R-62", + "name": "Road-150 Red, 62", + "description": "The product called \"Road-150 Red, 62\"", + "price": 3578.27 + }, + { + "id": "6EB9F7AC-7FB0-4D8C-8D3F-76A735A3CB9A", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18B-40", + "name": "Mountain-500 Black, 40", + "description": "The product called \"Mountain-500 Black, 40\"", + "price": 539.99 + }, + { + "id": "707106D2-0687-4217-AD2C-A6B828DFE075", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-R762", + "name": "ML Road Rear Wheel", + "description": "The product called \"ML Road Rear Wheel\"", + "price": 275.385 + }, + { + "id": "794ACC61-01E9-49BF-B150-1D02EE01D76F", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38B-48", + "name": "LL Road Frame - Black, 48", + "description": "The product called \"LL Road Frame - Black, 48\"", + "price": 337.22 + }, + { + "id": "86FD9250-4BD5-42D2-B941-1C1865A6A65E", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67U-58", + "name": "LL Touring Frame - Blue, 58", + "description": "The product called \"LL Touring Frame - Blue, 58\"", + "price": 333.42 + }, + { + "id": "8826E4D4-36FF-42AD-A33F-0E7794215158", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38B-58", + "name": "LL Road Frame - Black, 58", + "description": "The product called \"LL Road Frame - Black, 58\"", + "price": 337.22 + }, + { + "id": "894D03FA-1A4A-4FA4-9A0B-C3169EBB5674", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72Y-44", + "name": "ML Road Frame-W - Yellow, 44", + "description": "The product called \"ML Road Frame-W - Yellow, 44\"", + "price": 594.83 + }, + { + "id": "8BAA2AFB-CAE5-4A96-ABB2-46EDF9B5680E", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38B-60", + "name": "LL Road Frame - Black, 60", + "description": "The product called \"LL Road Frame - Black, 60\"", + "price": 337.22 + }, + { + "id": "8DB727BC-BE6B-4472-93F9-977B927D0C36", + "categoryId": "ACCC1FC1-7601-4F7A-AFA7-29C892F0FBE3", + "sku": "CA-1098", + "name": "AWC Logo Cap", + "description": "The product called \"AWC Logo Cap\"", + "price": 8.99 + }, + { + "id": "9183E546-A94B-4B7F-845B-A53E0EF5C626", + "categoryId": "973B839C-BF5D-485D-9D17-863C59B262E3", + "sku": "FK-5136", + "name": "ML Fork", + "description": "The product called \"ML Fork\"", + "price": 175.49 + }, + { + "id": "9351199A-B781-482D-80BE-2C11394002E5", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72Y-38", + "name": "ML Road Frame-W - Yellow, 38", + "description": "The product called \"ML Road Frame-W - Yellow, 38\"", + "price": 594.83 + }, + { + "id": "93A037C1-7135-4544-A688-3A3A75F25D0E", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94B-42", + "name": "HL Mountain Frame - Black, 42", + "description": "The product called \"HL Mountain Frame - Black, 42\"", + "price": 1349.6 + }, + { + "id": "98324A24-9D56-4662-93A5-9A7370E7EE5A", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-T762", + "name": "ML Touring Seat/Saddle", + "description": "The product called \"ML Touring Seat/Saddle\"", + "price": 39.14 + }, + { + "id": "9FEC8F06-D741-42EC-AF1D-E2F83BABC9F5", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63S-38", + "name": "ML Mountain Frame-W - Silver, 38", + "description": "The product called \"ML Mountain Frame-W - Silver, 38\"", + "price": 364.09 + }, + { + "id": "A042C88C-B060-4A64-B314-ED92124047E5", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-R853", + "name": "HL Road Pedal", + "description": "The product called \"HL Road Pedal\"", + "price": 80.99 + }, + { + "id": "ABB81D0E-4744-44EC-8AAB-FB3962FD2AF7", + "categoryId": "32A9A8E6-7004-4B24-9C2A-BB3E93B9E6BD", + "sku": "GL-F110-L", + "name": "Full-Finger Gloves, L", + "description": "The product called \"Full-Finger Gloves, L\"", + "price": 37.99 + }, + { + "id": "AFBE0496-C372-4885-B509-507B93027174", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18S-44", + "name": "Mountain-500 Silver, 44", + "description": "The product called \"Mountain-500 Silver, 44\"", + "price": 564.99 + }, + { + "id": "B03973CE-FAAD-4BE2-84FF-5BA5C751B6D0", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92B-44", + "name": "HL Road Frame - Black, 44", + "description": "The product called \"HL Road Frame - Black, 44\"", + "price": 1431.5 + }, + { + "id": "B39A06DD-3A51-470E-8253-8D6ACB3EA102", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21S-48", + "name": "LL Mountain Frame - Silver, 48", + "description": "The product called \"LL Mountain Frame - Silver, 48\"", + "price": 264.05 + }, + { + "id": "B73FFF5D-37A0-4A29-A42C-D91CD6743593", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50R-52", + "name": "Road-650 Red, 52", + "description": "The product called \"Road-650 Red, 52\"", + "price": 782.99 + }, + { + "id": "B8587D85-224F-4252-9521-A1763D63AEC2", + "categoryId": "ECEEC6AC-3CF1-41A6-8430-A1255F355BB5", + "sku": "RB-9231", + "name": "Rear Brakes", + "description": "The product called \"Rear Brakes\"", + "price": 106.5 + }, + { + "id": "BCD77A3D-9FF1-4CE4-9327-4C2A41BA9F0F", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-T852", + "name": "Touring Pedal", + "description": "The product called \"Touring Pedal\"", + "price": 80.99 + }, + { + "id": "C7B411C0-31F7-4634-B62F-ED349027EFE0", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-W890-L", + "name": "Women's Mountain Shorts, L", + "description": "The product called \"Women's Mountain Shorts, L\"", + "price": 69.99 + }, + { + "id": "CEA9FD38-517E-474B-A5B1-B17BF1753F9C", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67Y-50", + "name": "LL Touring Frame - Yellow, 50", + "description": "The product called \"LL Touring Frame - Yellow, 50\"", + "price": 333.42 + }, + { + "id": "D17F948A-2316-4E2E-8D89-973C92FAD9ED", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63S-42", + "name": "ML Mountain Frame-W - Silver, 42", + "description": "The product called \"ML Mountain Frame-W - Silver, 42\"", + "price": 364.09 + }, + { + "id": "D616598D-3159-4616-BF9D-FD316BF07224", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R68R-60", + "name": "Road-450 Red, 60", + "description": "The product called \"Road-450 Red, 60\"", + "price": 1457.99 + }, + { + "id": "D9FCCC86-10FD-47E9-B68B-F0DFE758AA0E", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-M762", + "name": "ML Mountain Front Wheel", + "description": "The product called \"ML Mountain Front Wheel\"", + "price": 209.025 + }, + { + "id": "DA96F0D0-84C7-42C3-BE74-FEB39BD60EF5", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18Y-62", + "name": "Touring-3000 Yellow, 62", + "description": "The product called \"Touring-3000 Yellow, 62\"", + "price": 742.35 + }, + { + "id": "DC8209E8-151E-425C-B7D9-7F082B66E39D", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-M940", + "name": "HL Mountain Seat/Saddle", + "description": "The product called \"HL Mountain Seat/Saddle\"", + "price": 52.64 + }, + { + "id": "DF94F21F-4CDB-4E49-B67B-CAD318A31C4A", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82B-44", + "name": "Mountain-100 Black, 44", + "description": "The product called \"Mountain-100 Black, 44\"", + "price": 3374.99 + }, + { + "id": "E08E4507-9666-411B-AAC4-519C00596B0A", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-R092", + "name": "LL Road Tire", + "description": "The product called \"LL Road Tire\"", + "price": 21.49 + }, + { + "id": "E223E34D-E0D0-4DFA-AB7D-8E72F94F2202", + "categoryId": "975E2A45-DA17-45CE-B65E-575A19334EB2", + "sku": "FD-2342", + "name": "Front Derailleur", + "description": "The product called \"Front Derailleur\"", + "price": 91.49 + }, + { + "id": "E54D5E31-073F-4D37-8400-E63A2994C92C", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-R762", + "name": "ML Road Front Wheel", + "description": "The product called \"ML Road Front Wheel\"", + "price": 248.385 + }, + { + "id": "E5A67B5B-B190-45CB-A9E4-BE3F6BD49214", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72R-48", + "name": "ML Road Frame - Red, 48", + "description": "The product called \"ML Road Frame - Red, 48\"", + "price": 594.83 + }, + { + "id": "E60D6D23-0151-4B7E-BC56-598B9FEE026B", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79Y-54", + "name": "Touring-1000 Yellow, 54", + "description": "The product called \"Touring-1000 Yellow, 54\"", + "price": 2384.07 + }, + { + "id": "E8767BC9-D6BA-47FC-9842-3511468869B6", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M47B-48", + "name": "Mountain-300 Black, 48", + "description": "The product called \"Mountain-300 Black, 48\"", + "price": 1079.99 + }, + { + "id": "EDCB55C5-4CF5-424F-9083-310F940879FA", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T44U-46", + "name": "Touring-2000 Blue, 46", + "description": "The product called \"Touring-2000 Blue, 46\"", + "price": 1214.85 + }, + { + "id": "EE40F7FD-AB2C-4589-B54D-BEBACB3B083E", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82B-48", + "name": "Mountain-100 Black, 48", + "description": "The product called \"Mountain-100 Black, 48\"", + "price": 3374.99 + }, + { + "id": "EEE4159B-F224-4C02-B578-2F398229592D", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38B-52", + "name": "LL Road Frame - Black, 52", + "description": "The product called \"LL Road Frame - Black, 52\"", + "price": 337.22 + }, + { + "id": "F2447558-7C01-442E-A7BC-B6D5D8AE1070", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82B-38", + "name": "Mountain-100 Black, 38", + "description": "The product called \"Mountain-100 Black, 38\"", + "price": 3374.99 + }, + { + "id": "F25B4447-9094-42DB-8244-186A279E461C", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50B-58", + "name": "Road-650 Black, 58", + "description": "The product called \"Road-650 Black, 58\"", + "price": 782.99 + }, + { + "id": "F58F50FB-BE83-4AE1-ACF0-662F702B2E5A", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89R-48", + "name": "Road-250 Red, 48", + "description": "The product called \"Road-250 Red, 48\"", + "price": 2443.35 + }, + { + "id": "F59ECC09-CAA5-4D3C-87A7-16945A92EA2D", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-W890-S", + "name": "Women's Mountain Shorts, S", + "description": "The product called \"Women's Mountain Shorts, S\"", + "price": 69.99 + }, + { + "id": "F6628734-A209-46A2-9010-0F19E7D3F3D3", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-R908", + "name": "ML Road Seat/Saddle", + "description": "The product called \"ML Road Seat/Saddle\"", + "price": 39.14 + }, + { + "id": "FADA3DBE-28DC-4FFA-823E-99332AD2EA0C", + "categoryId": "11EF8851-816A-49E2-9D5C-8D17AB82C5FF", + "sku": "LT-H902", + "name": "Headlights - Dual-Beam", + "description": "The product called \"Headlights - Dual-Beam\"", + "price": 34.99 + }, + { + "id": "FB118699-4C89-493B-B0AB-DA517935773E", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-T312", + "name": "LL Touring Seat/Saddle", + "description": "The product called \"LL Touring Seat/Saddle\"", + "price": 27.12 + }, + { + "id": "FC0B659C-C1EF-41F3-AFE2-F87C7F43AD48", + "categoryId": "AB952F9F-5ABA-4251-BC2D-AFF8DF412A4A", + "sku": "HS-0296", + "name": "LL Headset", + "description": "The product called \"LL Headset\"", + "price": 34.2 + }, + { + "id": "FCF95DBC-BBAD-467B-9639-FC6E4EC42B4C", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-R347", + "name": "LL Road Pedal", + "description": "The product called \"LL Road Pedal\"", + "price": 40.49 + }, + { + "id": "056C459F-DA40-475E-B7BE-B87B6DB39D33", + "categoryId": "9268EA12-29BA-404B-B514-E4737DB3BFCB", + "sku": "SB-M891-S", + "name": "Men's Bib-Shorts, S", + "description": "The product called \"Men's Bib-Shorts, S\"", + "price": 89.99 + }, + { + "id": "0B77351B-8F31-45D2-AECC-85BABD03B24E", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98Y-50", + "name": "HL Touring Frame - Yellow, 50", + "description": "The product called \"HL Touring Frame - Yellow, 50\"", + "price": 1003.91 + }, + { + "id": "14912B0B-EA77-47B8-8F1C-C8E4BE859D7C", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21S-42", + "name": "LL Mountain Frame - Silver, 42", + "description": "The product called \"LL Mountain Frame - Silver, 42\"", + "price": 264.05 + }, + { + "id": "1BEAE2B0-134A-4780-9A7A-5FA17EADD513", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92R-48", + "name": "HL Road Frame - Red, 48", + "description": "The product called \"HL Road Frame - Red, 48\"", + "price": 1431.5 + }, + { + "id": "2595584F-EA4E-4D45-948E-99A17AF8C519", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R19B-48", + "name": "Road-750 Black, 48", + "description": "The product called \"Road-750 Black, 48\"", + "price": 539.99 + }, + { + "id": "28A93A52-553C-4755-A2C4-07C1F5BD30F5", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94B-46", + "name": "HL Mountain Frame - Black, 46", + "description": "The product called \"HL Mountain Frame - Black, 46\"", + "price": 1349.6 + }, + { + "id": "29663491-D2E9-47B4-83AE-D9459B6B5B67", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TT-T092", + "name": "Touring Tire Tube", + "description": "The product called \"Touring Tire Tube\"", + "price": 4.99 + }, + { + "id": "2EE56307-0398-465E-A340-1C5FB1C85648", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18S-42", + "name": "Mountain-500 Silver, 42", + "description": "The product called \"Mountain-500 Silver, 42\"", + "price": 564.99 + }, + { + "id": "311D60FC-9EB9-4194-B594-1E5BD87CCF81", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38R-62", + "name": "LL Road Frame - Red, 62", + "description": "The product called \"LL Road Frame - Red, 62\"", + "price": 337.22 + }, + { + "id": "3F105575-8677-42F9-8E1F-76E4B450F136", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79Y-46", + "name": "Touring-1000 Yellow, 46", + "description": "The product called \"Touring-1000 Yellow, 46\"", + "price": 2384.07 + }, + { + "id": "3F3E4045-AC4D-4D28-99D5-6C9C53F1DEAF", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21B-40", + "name": "LL Mountain Frame - Black, 40", + "description": "The product called \"LL Mountain Frame - Black, 40\"", + "price": 249.79 + }, + { + "id": "4424AA2A-CC8D-4471-9478-21E91185593C", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92B-48", + "name": "HL Road Frame - Black, 48", + "description": "The product called \"HL Road Frame - Black, 48\"", + "price": 1431.5 + }, + { + "id": "49E5C64F-B689-4C0B-9E2C-5DFF006B929D", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63S-40", + "name": "ML Mountain Frame-W - Silver, 40", + "description": "The product called \"ML Mountain Frame-W - Silver, 40\"", + "price": 364.09 + }, + { + "id": "520E3E6B-95F1-4258-9F74-E434848E88B2", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38R-48", + "name": "LL Road Frame - Red, 48", + "description": "The product called \"LL Road Frame - Red, 48\"", + "price": 337.22 + }, + { + "id": "5308BAE7-B0CB-4883-9A93-192CB10DC94F", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18U-44", + "name": "Touring-3000 Blue, 44", + "description": "The product called \"Touring-3000 Blue, 44\"", + "price": 742.35 + }, + { + "id": "58B0F878-2619-4225-B9B1-9C6C4FFF9C17", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38B-44", + "name": "LL Road Frame - Black, 44", + "description": "The product called \"LL Road Frame - Black, 44\"", + "price": 337.22 + }, + { + "id": "5996B5E0-6EC7-4CB7-A924-7B5A053AE980", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-M236", + "name": "LL Mountain Seat/Saddle", + "description": "The product called \"LL Mountain Seat/Saddle\"", + "price": 27.12 + }, + { + "id": "5BFADECD-2240-4480-9485-1256D1D60EA8", + "categoryId": "32A9A8E6-7004-4B24-9C2A-BB3E93B9E6BD", + "sku": "GL-F110-S", + "name": "Full-Finger Gloves, S", + "description": "The product called \"Full-Finger Gloves, S\"", + "price": 37.99 + }, + { + "id": "5C30FF31-CAB7-4A99-8FD6-D610F58AC4BA", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M38S-42", + "name": "Mountain-400-W Silver, 42", + "description": "The product called \"Mountain-400-W Silver, 42\"", + "price": 769.49 + }, + { + "id": "601A5234-644D-4B83-9FDB-326C22C1051D", + "categoryId": "14A1AD5D-59EA-4B63-A189-67B077783B0E", + "sku": "HL-U509-B", + "name": "Sport-100 Helmet, Blue", + "description": "The product called \"Sport-100 Helmet, Blue\"", + "price": 34.99 + }, + { + "id": "61246D01-7C38-489E-9F49-A526679B568F", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67U-50", + "name": "LL Touring Frame - Blue, 50", + "description": "The product called \"LL Touring Frame - Blue, 50\"", + "price": 333.42 + }, + { + "id": "6401B68F-924A-4B2E-AC9E-5660AEA0E848", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "LJ-0192-L", + "name": "Long-Sleeve Logo Jersey, L", + "description": "The product called \"Long-Sleeve Logo Jersey, L\"", + "price": 49.99 + }, + { + "id": "6E3AA511-67DF-4EAD-8F0C-4C9F91F7D335", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18Y-50", + "name": "Touring-3000 Yellow, 50", + "description": "The product called \"Touring-3000 Yellow, 50\"", + "price": 742.35 + }, + { + "id": "71BDFE67-6499-4A8E-9CCA-9E9AF7D92A7A", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18U-54", + "name": "Touring-3000 Blue, 54", + "description": "The product called \"Touring-3000 Blue, 54\"", + "price": 742.35 + }, + { + "id": "744A624B-E4C2-429E-8A69-DC3B57682BD5", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21B-42", + "name": "LL Mountain Frame - Black, 42", + "description": "The product called \"LL Mountain Frame - Black, 42\"", + "price": 249.79 + }, + { + "id": "7AD4F00E-BB64-4B02-AC6B-0D5F04B01CAB", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R64Y-38", + "name": "Road-550-W Yellow, 38", + "description": "The product called \"Road-550-W Yellow, 38\"", + "price": 1120.49 + }, + { + "id": "7EF2B766-E966-4809-B568-372823002877", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63B-40", + "name": "ML Mountain Frame - Black, 40", + "description": "The product called \"ML Mountain Frame - Black, 40\"", + "price": 348.76 + }, + { + "id": "824D58CA-ECCA-4E72-965C-66D3A5C0C67C", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67Y-58", + "name": "LL Touring Frame - Yellow, 58", + "description": "The product called \"LL Touring Frame - Yellow, 58\"", + "price": 333.42 + }, + { + "id": "840E2138-4265-4AC8-8514-AC0B9C98597C", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67U-44", + "name": "LL Touring Frame - Blue, 44", + "description": "The product called \"LL Touring Frame - Blue, 44\"", + "price": 333.42 + }, + { + "id": "8A4C4A7F-6EE1-4436-89E3-80AA2D8A1154", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-R623", + "name": "LL Road Rear Wheel", + "description": "The product called \"LL Road Rear Wheel\"", + "price": 112.565 + }, + { + "id": "9363838B-2D13-48E8-986D-C9625BE5AB26", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50B-60", + "name": "Road-650 Black, 60", + "description": "The product called \"Road-650 Black, 60\"", + "price": 782.99 + }, + { + "id": "987E39AC-6C62-4717-9929-E9BDFF9902ED", + "categoryId": "34340561-3D26-4F33-B6AD-09260FC811D6", + "sku": "BB-8107", + "name": "ML Bottom Bracket", + "description": "The product called \"ML Bottom Bracket\"", + "price": 101.24 + }, + { + "id": "9DB28F2B-ADC8-40A2-A677-B0AAFC32CAC8", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82S-48", + "name": "Mountain-100 Silver, 48", + "description": "The product called \"Mountain-100 Silver, 48\"", + "price": 3399.99 + }, + { + "id": "9E5C74FD-F685-45AE-A799-D67EFB5C28A1", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R79Y-40", + "name": "Road-350-W Yellow, 40", + "description": "The product called \"Road-350-W Yellow, 40\"", + "price": 1700.99 + }, + { + "id": "A2E169C8-0916-4CAD-9C7B-FBAF463D0DB3", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72Y-42", + "name": "ML Road Frame-W - Yellow, 42", + "description": "The product called \"ML Road Frame-W - Yellow, 42\"", + "price": 594.83 + }, + { + "id": "A6040C40-906B-4A87-9E2C-683A8037A1C3", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92B-52", + "name": "HL Road Frame - Black, 52", + "description": "The product called \"HL Road Frame - Black, 52\"", + "price": 1431.5 + }, + { + "id": "A875BC33-C4AC-4D2B-B018-9FF4672A2BB9", + "categoryId": "32A9A8E6-7004-4B24-9C2A-BB3E93B9E6BD", + "sku": "GL-H102-L", + "name": "Half-Finger Gloves, L", + "description": "The product called \"Half-Finger Gloves, L\"", + "price": 24.49 + }, + { + "id": "ACD4ABE3-82D8-4447-B126-2DE03B7DD106", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M38S-40", + "name": "Mountain-400-W Silver, 40", + "description": "The product called \"Mountain-400-W Silver, 40\"", + "price": 769.49 + }, + { + "id": "B10065F8-543A-49E7-BFE6-3D19B0BE5670", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M68S-46", + "name": "Mountain-200 Silver, 46", + "description": "The product called \"Mountain-200 Silver, 46\"", + "price": 2319.99 + }, + { + "id": "B3847F90-FDF3-4529-B7D0-04FE6F94BFB3", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18S-48", + "name": "Mountain-500 Silver, 48", + "description": "The product called \"Mountain-500 Silver, 48\"", + "price": 564.99 + }, + { + "id": "BB21E6EF-104A-420B-B9C5-2084118E5A2F", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M68S-42", + "name": "Mountain-200 Silver, 42", + "description": "The product called \"Mountain-200 Silver, 42\"", + "price": 2319.99 + }, + { + "id": "BF381234-799A-4B1A-BD4B-B55891CC5907", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T44U-50", + "name": "Touring-2000 Blue, 50", + "description": "The product called \"Touring-2000 Blue, 50\"", + "price": 1214.85 + }, + { + "id": "BF87ACE3-C52B-44EA-9871-4A6497B3AF9F", + "categoryId": "7FF64215-1F7A-4CDF-9BA1-AD6ADC6B5D1C", + "sku": "PU-0452", + "name": "Minipump", + "description": "The product called \"Minipump\"", + "price": 19.99 + }, + { + "id": "C310A68D-DBF2-421A-91CA-F09A3B8A1AAA", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72R-52", + "name": "ML Road Frame - Red, 52", + "description": "The product called \"ML Road Frame - Red, 52\"", + "price": 594.83 + }, + { + "id": "C6941C95-C463-4F66-BE5F-8CA9C5F7FD91", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89R-52", + "name": "Road-250 Red, 52", + "description": "The product called \"Road-250 Red, 52\"", + "price": 2443.35 + }, + { + "id": "C7BE1762-AC9D-4239-BD15-F3096B08AFA9", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-M423", + "name": "LL Mountain Front Wheel", + "description": "The product called \"LL Mountain Front Wheel\"", + "price": 60.745 + }, + { + "id": "CAC12FD1-C2ED-4B75-9199-86EB2044DB0D", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63B-44", + "name": "ML Mountain Frame - Black, 44", + "description": "The product called \"ML Mountain Frame - Black, 44\"", + "price": 348.76 + }, + { + "id": "CD5FF4D6-7D2D-4BD4-9319-CB38C1939D96", + "categoryId": "11EF8851-816A-49E2-9D5C-8D17AB82C5FF", + "sku": "LT-T990", + "name": "Taillights - Battery-Powered", + "description": "The product called \"Taillights - Battery-Powered\"", + "price": 13.99 + }, + { + "id": "DDD64AA0-30DC-4DC1-BCDC-2882A0FD178C", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18U-50", + "name": "Touring-3000 Blue, 50", + "description": "The product called \"Touring-3000 Blue, 50\"", + "price": 742.35 + }, + { + "id": "DFE5521E-40C6-4A58-8E8D-5FC1BE5EC0FE", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82S-44", + "name": "Mountain-100 Silver, 44", + "description": "The product called \"Mountain-100 Silver, 44\"", + "price": 3399.99 + }, + { + "id": "E2CCAF6F-7AB5-4086-86A3-A50B3E6EF101", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98Y-60", + "name": "HL Touring Frame - Yellow, 60", + "description": "The product called \"HL Touring Frame - Yellow, 60\"", + "price": 1003.91 + }, + { + "id": "E2FD2420-B084-4764-8BC4-94574DFF1AC6", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R68R-48", + "name": "Road-450 Red, 48", + "description": "The product called \"Road-450 Red, 48\"", + "price": 1457.99 + }, + { + "id": "E681778F-8359-468B-98F9-4D325D6C377F", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "SJ-0194-S", + "name": "Short-Sleeve Classic Jersey, S", + "description": "The product called \"Short-Sleeve Classic Jersey, S\"", + "price": 53.99 + }, + { + "id": "E9FCF7AC-1F45-4857-9E75-BC30A7C7C27B", + "categoryId": "C0EB227A-55A9-498B-8E21-F39EC5088143", + "sku": "CL-9009", + "name": "Bike Wash - Dissolver", + "description": "The product called \"Bike Wash - Dissolver\"", + "price": 7.95 + }, + { + "id": "EB793BFC-82A4-4EF4-BB2A-4FD218DD1843", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-M897-L", + "name": "Men's Sports Shorts, L", + "description": "The product called \"Men's Sports Shorts, L\"", + "price": 59.99 + }, + { + "id": "EF3F4DC1-5F73-4234-B10E-6608F4DC937A", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M68S-38", + "name": "Mountain-200 Silver, 38", + "description": "The product called \"Mountain-200 Silver, 38\"", + "price": 2319.99 + }, + { + "id": "F42672DA-1B19-463B-B49D-AC4EA2E1F77C", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89B-48", + "name": "Road-250 Black, 48", + "description": "The product called \"Road-250 Black, 48\"", + "price": 2443.35 + }, + { + "id": "F5FB0386-C6AC-40AE-9342-7AFB832233A8", + "categoryId": "27A716B2-6F81-4A2C-B7E9-0B2AF5D8E51A", + "sku": "LO-C100", + "name": "Cable Lock", + "description": "The product called \"Cable Lock\"", + "price": 25.0 + }, + { + "id": "F741B78B-36F0-42E9-A26A-FAE908E0FB3A", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-T905", + "name": "Touring Rear Wheel", + "description": "The product called \"Touring Rear Wheel\"", + "price": 245.01 + }, + { + "id": "FA06B762-D602-4235-8F77-D8AFB0D3D050", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67U-54", + "name": "LL Touring Frame - Blue, 54", + "description": "The product called \"LL Touring Frame - Blue, 54\"", + "price": 333.42 + }, + { + "id": "FD48A179-6CF5-45F2-8605-9DA19B9D4409", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R93R-44", + "name": "Road-150 Red, 44", + "description": "The product called \"Road-150 Red, 44\"", + "price": 3578.27 + }, + { + "id": "FE8FFBD3-99AE-4ECF-AA53-D1304D941EC7", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M47B-40", + "name": "Mountain-300 Black, 40", + "description": "The product called \"Mountain-300 Black, 40\"", + "price": 1079.99 + }, + { + "id": "027D0B9A-F9D9-4C96-8213-C8546C4AAE71", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-R581", + "name": "LL Road Seat/Saddle", + "description": "The product called \"LL Road Seat/Saddle\"", + "price": 27.12 + }, + { + "id": "08CF5494-D064-40CF-952B-E33ED9CE9297", + "categoryId": "9268EA12-29BA-404B-B514-E4737DB3BFCB", + "sku": "SB-M891-M", + "name": "Men's Bib-Shorts, M", + "description": "The product called \"Men's Bib-Shorts, M\"", + "price": 89.99 + }, + { + "id": "0990C3D9-4EC2-4272-ADB6-9481CA12F5F6", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94B-38", + "name": "HL Mountain Frame - Black, 38", + "description": "The product called \"HL Mountain Frame - Black, 38\"", + "price": 1349.6 + }, + { + "id": "0F124781-C991-48A9-ACF2-249771D44029", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M68B-42", + "name": "Mountain-200 Black, 42", + "description": "The product called \"Mountain-200 Black, 42\"", + "price": 2294.99 + }, + { + "id": "1E0D3EBA-563D-4DA1-8D6C-FE9C7A63EE2B", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-R820", + "name": "HL Road Front Wheel", + "description": "The product called \"HL Road Front Wheel\"", + "price": 330.06 + }, + { + "id": "25B35002-7F61-45E3-AA55-80A743C3BC36", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "FW-R623", + "name": "LL Road Front Wheel", + "description": "The product called \"LL Road Front Wheel\"", + "price": 85.565 + }, + { + "id": "26E6C049-667F-4463-AF1D-660953231165", + "categoryId": "8797AB0F-A9A3-475D-925E-56AC73DC206E", + "sku": "CH-0234", + "name": "Chain", + "description": "The product called \"Chain\"", + "price": 20.24 + }, + { + "id": "28A865D5-647E-46B5-B309-CA2B2F524E37", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98U-60", + "name": "HL Touring Frame - Blue, 60", + "description": "The product called \"HL Touring Frame - Blue, 60\"", + "price": 1003.91 + }, + { + "id": "290B4594-95BE-47C5-863A-4EFAAFC0AED7", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-M602", + "name": "ML Mountain Tire", + "description": "The product called \"ML Mountain Tire\"", + "price": 29.99 + }, + { + "id": "295ABC00-9080-479C-9733-A9BE712D7A18", + "categoryId": "C48B4EF4-D352-4CD2-BCB8-CE89B7DFA642", + "sku": "SO-R809-L", + "name": "Racing Socks, L", + "description": "The product called \"Racing Socks, L\"", + "price": 8.99 + }, + { + "id": "2BBCE73F-9D1D-4BE1-808C-8B174D0DA1A2", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21B-52", + "name": "LL Mountain Frame - Black, 52", + "description": "The product called \"LL Mountain Frame - Black, 52\"", + "price": 249.79 + }, + { + "id": "2CE4EFA7-5DC6-4D3E-ACB2-B7DDE4518408", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92B-58", + "name": "HL Road Frame - Black, 58", + "description": "The product called \"HL Road Frame - Black, 58\"", + "price": 1431.5 + }, + { + "id": "332C8377-F7B5-44C2-8DFC-B374294FD9B2", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92R-62", + "name": "HL Road Frame - Red, 62", + "description": "The product called \"HL Road Frame - Red, 62\"", + "price": 1431.5 + }, + { + "id": "3B52D15D-DF6C-4042-BA15-2EFEA8A2F852", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92B-62", + "name": "HL Road Frame - Black, 62", + "description": "The product called \"HL Road Frame - Black, 62\"", + "price": 1431.5 + }, + { + "id": "3CE3E061-88E1-4430-BAC7-809B285FC702", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-M897-M", + "name": "Men's Sports Shorts, M", + "description": "The product called \"Men's Sports Shorts, M\"", + "price": 59.99 + }, + { + "id": "3FA9E0D9-E6E9-429D-9E24-7DAFE9B99A2C", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92R-58", + "name": "HL Road Frame - Red, 58", + "description": "The product called \"HL Road Frame - Red, 58\"", + "price": 1431.5 + }, + { + "id": "435D4B82-D557-4752-B825-D28767FB32D3", + "categoryId": "AA5A82D4-914C-4132-8C08-E7B75DCE3428", + "sku": "CS-4759", + "name": "LL Crankset", + "description": "The product called \"LL Crankset\"", + "price": 175.49 + }, + { + "id": "4B0848F8-7BF5-4DB9-84A7-C4D69F2E3E8E", + "categoryId": "34340561-3D26-4F33-B6AD-09260FC811D6", + "sku": "BB-7421", + "name": "LL Bottom Bracket", + "description": "The product called \"LL Bottom Bracket\"", + "price": 53.99 + }, + { + "id": "4DA12D36-495E-4DCA-95B0-F18CAA099779", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82S-42", + "name": "Mountain-100 Silver, 42", + "description": "The product called \"Mountain-100 Silver, 42\"", + "price": 3399.99 + }, + { + "id": "4F9FC42A-F43F-4C13-92FC-ADF701F48C36", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R79Y-44", + "name": "Road-350-W Yellow, 44", + "description": "The product called \"Road-350-W Yellow, 44\"", + "price": 1700.99 + }, + { + "id": "50DC9B64-03B8-49AB-9DB4-75D12B3180D8", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M63B-48", + "name": "ML Mountain Frame - Black, 48", + "description": "The product called \"ML Mountain Frame - Black, 48\"", + "price": 348.76 + }, + { + "id": "56560B7B-3AC6-4E07-8825-4266A7C98CFE", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18B-52", + "name": "Mountain-500 Black, 52", + "description": "The product called \"Mountain-500 Black, 52\"", + "price": 539.99 + }, + { + "id": "5BC9F76B-7FE9-4DD9-A672-2C5E802B2672", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "SJ-0194-L", + "name": "Short-Sleeve Classic Jersey, L", + "description": "The product called \"Short-Sleeve Classic Jersey, L\"", + "price": 53.99 + }, + { + "id": "64B3F15E-3E21-4ECD-9013-E50ABD324337", + "categoryId": "C80E3277-604C-4C6D-85AE-FCB237C08751", + "sku": "RW-M423", + "name": "LL Mountain Rear Wheel", + "description": "The product called \"LL Mountain Rear Wheel\"", + "price": 87.745 + }, + { + "id": "668E6FCE-03E9-49E7-AC33-1B17FEEF5E60", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M18S-40", + "name": "Mountain-500 Silver, 40", + "description": "The product called \"Mountain-500 Silver, 40\"", + "price": 564.99 + }, + { + "id": "6FB5B2D5-5725-4998-9B6C-2FF2B7A3E3E0", + "categoryId": "32A9A8E6-7004-4B24-9C2A-BB3E93B9E6BD", + "sku": "GL-H102-M", + "name": "Half-Finger Gloves, M", + "description": "The product called \"Half-Finger Gloves, M\"", + "price": 24.49 + }, + { + "id": "7236DDB5-CFE0-4D3D-8FE5-799B398396B1", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50B-48", + "name": "Road-650 Black, 48", + "description": "The product called \"Road-650 Black, 48\"", + "price": 782.99 + }, + { + "id": "7355D821-E33B-410B-AE64-D5A535F767EB", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-M243", + "name": "LL Mountain Handlebars", + "description": "The product called \"LL Mountain Handlebars\"", + "price": 44.54 + }, + { + "id": "751115E7-BD5E-45C7-932B-E9DDE9D62579", + "categoryId": "973B839C-BF5D-485D-9D17-863C59B262E3", + "sku": "FK-9939", + "name": "HL Fork", + "description": "The product called \"HL Fork\"", + "price": 229.49 + }, + { + "id": "78E7D28A-2D53-40DC-9ED2-8E2841820DEB", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50B-62", + "name": "Road-650 Black, 62", + "description": "The product called \"Road-650 Black, 62\"", + "price": 782.99 + }, + { + "id": "7BAA49C9-21B5-4EEF-9F6B-BCD6DA7C2239", + "categoryId": "26C74104-40BC-4541-8EF5-9892F7F03D72", + "sku": "SE-R995", + "name": "HL Road Seat/Saddle", + "description": "The product called \"HL Road Seat/Saddle\"", + "price": 52.64 + }, + { + "id": "7EA0EEEB-824E-42E9-B787-019219CE4466", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T79U-54", + "name": "Touring-1000 Blue, 54", + "description": "The product called \"Touring-1000 Blue, 54\"", + "price": 2384.07 + }, + { + "id": "829B2717-0D74-43D3-BBD8-27CFDEF5ACA1", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94B-48", + "name": "HL Mountain Frame - Black, 48", + "description": "The product called \"HL Mountain Frame - Black, 48\"", + "price": 1349.6 + }, + { + "id": "878C50F0-7E29-4D0D-A52E-6D8B063673E3", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89R-58", + "name": "Road-250 Red, 58", + "description": "The product called \"Road-250 Red, 58\"", + "price": 2443.35 + }, + { + "id": "8D3DCF87-D1ED-44DD-8DB8-085EB98C8A52", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R92R-52", + "name": "HL Road Frame - Red, 52", + "description": "The product called \"HL Road Frame - Red, 52\"", + "price": 1431.5 + }, + { + "id": "8FE13D26-469C-41FE-BD7E-0A856A82F95C", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94S-38", + "name": "HL Mountain Frame - Silver, 38", + "description": "The product called \"HL Mountain Frame - Silver, 38\"", + "price": 1364.5 + }, + { + "id": "906A453F-2B5E-469A-87B5-FFA531EE615D", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "PK-7098", + "name": "Patch Kit/8 Patches", + "description": "The product called \"Patch Kit/8 Patches\"", + "price": 2.29 + }, + { + "id": "90888587-BBBD-4632-8A48-5B979586DEE4", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50R-62", + "name": "Road-650 Red, 62", + "description": "The product called \"Road-650 Red, 62\"", + "price": 782.99 + }, + { + "id": "9190229B-1372-4997-8F64-5B3E7A2459C5", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TT-M928", + "name": "Mountain Tire Tube", + "description": "The product called \"Mountain Tire Tube\"", + "price": 4.99 + }, + { + "id": "91AA100C-D092-4190-92A7-7C02410F04EA", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67Y-62", + "name": "LL Touring Frame - Yellow, 62", + "description": "The product called \"LL Touring Frame - Yellow, 62\"", + "price": 333.42 + }, + { + "id": "91D3C273-9E79-4395-B444-6D39BF9B2F4D", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T98U-46", + "name": "HL Touring Frame - Blue, 46", + "description": "The product called \"HL Touring Frame - Blue, 46\"", + "price": 1003.91 + }, + { + "id": "91E5405C-DC61-42CE-B900-0F46C94FBBA5", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R64Y-44", + "name": "Road-550-W Yellow, 44", + "description": "The product called \"Road-550-W Yellow, 44\"", + "price": 1120.49 + }, + { + "id": "92413209-8DA6-4661-9E11-26B55990BEB2", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38R-52", + "name": "LL Road Frame - Red, 52", + "description": "The product called \"LL Road Frame - Red, 52\"", + "price": 337.22 + }, + { + "id": "92DB7ABD-1C8E-458C-8828-9BFD1984B07D", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38B-62", + "name": "LL Road Frame - Black, 62", + "description": "The product called \"LL Road Frame - Black, 62\"", + "price": 337.22 + }, + { + "id": "94265B3D-7718-47F0-ADF7-64DEE36CAC41", + "categoryId": "629A8F3C-CFB0-4347-8DCC-505A4789876B", + "sku": "VE-C304-S", + "name": "Classic Vest, S", + "description": "The product called \"Classic Vest, S\"", + "price": 63.5 + }, + { + "id": "967155B3-9925-4FA3-84B0-B24CDA101C1B", + "categoryId": "629A8F3C-CFB0-4347-8DCC-505A4789876B", + "sku": "VE-C304-M", + "name": "Classic Vest, M", + "description": "The product called \"Classic Vest, M\"", + "price": 63.5 + }, + { + "id": "9851FE19-CCA4-4B94-B6AC-CCE579D7F693", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-M823", + "name": "HL Mountain Tire", + "description": "The product called \"HL Mountain Tire\"", + "price": 35.0 + }, + { + "id": "9C0320C4-124B-486A-BA98-B7B82933F324", + "categoryId": "F3FBB167-11D8-41E4-84B4-5AAA92B1E737", + "sku": "FR-T67U-62", + "name": "LL Touring Frame - Blue, 62", + "description": "The product called \"LL Touring Frame - Blue, 62\"", + "price": 333.42 + }, + { + "id": "9E5DD0E4-89B5-4300-BD49-87518EE9DB6A", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-R956", + "name": "HL Road Handlebars", + "description": "The product called \"HL Road Handlebars\"", + "price": 120.27 + }, + { + "id": "9E6692D7-57E1-4D35-ACD8-105D44A1073B", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18U-58", + "name": "Touring-3000 Blue, 58", + "description": "The product called \"Touring-3000 Blue, 58\"", + "price": 742.35 + }, + { + "id": "A1D803E1-B9DE-49B4-9E61-66F5C3CD679A", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-R504", + "name": "LL Road Handlebars", + "description": "The product called \"LL Road Handlebars\"", + "price": 44.54 + }, + { + "id": "A374B506-8D35-456B-8C63-BCE78B5083B8", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-W890-M", + "name": "Women's Mountain Shorts, M", + "description": "The product called \"Women's Mountain Shorts, M\"", + "price": 69.99 + }, + { + "id": "A6BB4603-7CD5-43DC-920A-2A2F55D52492", + "categoryId": "AB952F9F-5ABA-4251-BC2D-AFF8DF412A4A", + "sku": "HS-2451", + "name": "ML Headset", + "description": "The product called \"ML Headset\"", + "price": 102.29 + }, + { + "id": "A6F069C2-EF85-4B79-9CE2-03833343AD92", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "SJ-0194-M", + "name": "Short-Sleeve Classic Jersey, M", + "description": "The product called \"Short-Sleeve Classic Jersey, M\"", + "price": 53.99 + }, + { + "id": "ABDE32DD-FADD-4042-9278-0440B7B2F3E0", + "categoryId": "86F3CBAB-97A7-4D01-BABB-ADEFFFAED6B4", + "sku": "TI-M267", + "name": "LL Mountain Tire", + "description": "The product called \"LL Mountain Tire\"", + "price": 24.99 + }, + { + "id": "ACC683CB-6199-416E-AE64-7C10D0C72CF9", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-M562", + "name": "HL Mountain Pedal", + "description": "The product called \"HL Mountain Pedal\"", + "price": 80.99 + }, + { + "id": "AFED4FD0-17D1-4CD5-8639-13F15B043EC2", + "categoryId": "006A1D51-28DA-4956-A7FB-C0B2BF6360CA", + "sku": "WB-H098", + "name": "Water Bottle - 30 oz.", + "description": "The product called \"Water Bottle - 30 oz.\"", + "price": 4.99 + }, + { + "id": "B08450AA-413C-4663-A62D-7291A8ECF1F5", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R38R-44", + "name": "LL Road Frame - Red, 44", + "description": "The product called \"LL Road Frame - Red, 44\"", + "price": 337.22 + }, + { + "id": "B1AAF271-9DFA-4826-91A3-F3B4BFF49B1C", + "categoryId": "AA5A82D4-914C-4132-8C08-E7B75DCE3428", + "sku": "CS-6583", + "name": "ML Crankset", + "description": "The product called \"ML Crankset\"", + "price": 256.49 + }, + { + "id": "B267655B-A7C1-41E3-9682-21730E93FCB5", + "categoryId": "AA28AE74-D57C-4B23-B5F7-F919E1C5844E", + "sku": "TG-W091-L", + "name": "Women's Tights, L", + "description": "The product called \"Women's Tights, L\"", + "price": 74.99 + }, + { + "id": "B3217262-876C-4C29-A201-06101B710396", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21S-40", + "name": "LL Mountain Frame - Silver, 40", + "description": "The product called \"LL Mountain Frame - Silver, 40\"", + "price": 264.05 + }, + { + "id": "B35B87F4-5ADE-4ED4-9469-DF024AC4195D", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18Y-58", + "name": "Touring-3000 Yellow, 58", + "description": "The product called \"Touring-3000 Yellow, 58\"", + "price": 742.35 + }, + { + "id": "B3C8AE66-8E4B-4605-A78D-FF2A8C4EAD9A", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94S-46", + "name": "HL Mountain Frame - Silver, 46", + "description": "The product called \"HL Mountain Frame - Silver, 46\"", + "price": 1364.5 + }, + { + "id": "B6591222-0FB9-415F-8F2B-18B56A483AA1", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-R720", + "name": "ML Road Handlebars", + "description": "The product called \"ML Road Handlebars\"", + "price": 61.92 + }, + { + "id": "B8E30737-758B-49E0-A153-B210B80749F4", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21S-52", + "name": "LL Mountain Frame - Silver, 52", + "description": "The product called \"LL Mountain Frame - Silver, 52\"", + "price": 264.05 + }, + { + "id": "C0FBA4E8-B617-4889-B1A5-091D12783313", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M82B-42", + "name": "Mountain-100 Black, 42", + "description": "The product called \"Mountain-100 Black, 42\"", + "price": 3374.99 + }, + { + "id": "CE35E963-F6ED-4108-BC4B-6A3DD0557B47", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21B-44", + "name": "LL Mountain Frame - Black, 44", + "description": "The product called \"LL Mountain Frame - Black, 44\"", + "price": 249.79 + }, + { + "id": "D47E0CC9-28A0-40A5-AB90-BB29BDBB0578", + "categoryId": "C3C57C35-1D80-4EC5-AB12-46C57A017AFB", + "sku": "LJ-0192-X", + "name": "Long-Sleeve Logo Jersey, XL", + "description": "The product called \"Long-Sleeve Logo Jersey, XL\"", + "price": 49.99 + }, + { + "id": "D8CA2EB2-7532-4F74-9D1D-E8CCC7326604", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R50R-58", + "name": "Road-650 Red, 58", + "description": "The product called \"Road-650 Red, 58\"", + "price": 782.99 + }, + { + "id": "DB89A887-43E3-4D9C-8783-7F034ACD88C0", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T18Y-54", + "name": "Touring-3000 Yellow, 54", + "description": "The product called \"Touring-3000 Yellow, 54\"", + "price": 742.35 + }, + { + "id": "DE8C032F-472A-4FFE-A8AE-4C7FFAF06DA8", + "categoryId": "34340561-3D26-4F33-B6AD-09260FC811D6", + "sku": "BB-9108", + "name": "HL Bottom Bracket", + "description": "The product called \"HL Bottom Bracket\"", + "price": 121.49 + }, + { + "id": "E49AE44E-40AC-4FD8-A007-EEC046F02684", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M21S-44", + "name": "LL Mountain Frame - Silver, 44", + "description": "The product called \"LL Mountain Frame - Silver, 44\"", + "price": 264.05 + }, + { + "id": "E5C302BB-43AC-4E47-8355-F0D2165C394A", + "categoryId": "B5EF9CFA-FD22-4888-858D-2C8C5E4B2EFA", + "sku": "HB-M763", + "name": "ML Mountain Handlebars", + "description": "The product called \"ML Mountain Handlebars\"", + "price": 61.92 + }, + { + "id": "E5CEC513-A0F9-4437-B26D-A9FB28237554", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89B-44", + "name": "Road-250 Black, 44", + "description": "The product called \"Road-250 Black, 44\"", + "price": 2443.35 + }, + { + "id": "E78CEEF9-A87B-4612-8BD3-4E5DC8AC4700", + "categoryId": "75BF1ACB-168D-469C-9AA3-1FD26BB4EA4C", + "sku": "BK-T44U-54", + "name": "Touring-2000 Blue, 54", + "description": "The product called \"Touring-2000 Blue, 54\"", + "price": 1214.85 + }, + { + "id": "EC2ADE30-9132-4DFE-B8FE-D233DDFFAAB3", + "categoryId": "AE48F0AA-4F65-4734-A4CF-D48B8F82267F", + "sku": "BK-R89B-52", + "name": "Road-250 Black, 52", + "description": "The product called \"Road-250 Black, 52\"", + "price": 2443.35 + }, + { + "id": "EC65B816-A2A7-4245-B138-43C03F14C514", + "categoryId": "56400CF3-446D-4C3F-B9B2-68286DA3BB99", + "sku": "BK-M68B-46", + "name": "Mountain-200 Black, 46", + "description": "The product called \"Mountain-200 Black, 46\"", + "price": 2294.99 + }, + { + "id": "EF16A6FA-9BE2-4AF9-872A-299A9EA88D5F", + "categoryId": "32A9A8E6-7004-4B24-9C2A-BB3E93B9E6BD", + "sku": "GL-F110-M", + "name": "Full-Finger Gloves, M", + "description": "The product called \"Full-Finger Gloves, M\"", + "price": 37.99 + }, + { + "id": "EFD1F33B-94AE-4309-B6E6-F9CCC2B61278", + "categoryId": "11EF8851-816A-49E2-9D5C-8D17AB82C5FF", + "sku": "LT-H903", + "name": "Headlights - Weatherproof", + "description": "The product called \"Headlights - Weatherproof\"", + "price": 44.99 + }, + { + "id": "F07F8C10-4820-4C80-AAE2-1DDEC41E5A29", + "categoryId": "AB952F9F-5ABA-4251-BC2D-AFF8DF412A4A", + "sku": "HS-3479", + "name": "HL Headset", + "description": "The product called \"HL Headset\"", + "price": 124.73 + }, + { + "id": "F3012443-6317-4856-800A-6E108A5F8AE5", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94S-42", + "name": "HL Mountain Frame - Silver, 42", + "description": "The product called \"HL Mountain Frame - Silver, 42\"", + "price": 1364.5 + }, + { + "id": "F32990D7-F8E4-4ACD-AA8C-1F03D8299DE7", + "categoryId": "4F34E180-384D-42FC-AC10-FEC30227577F", + "sku": "PD-M282", + "name": "LL Mountain Pedal", + "description": "The product called \"LL Mountain Pedal\"", + "price": 40.49 + }, + { + "id": "F7078B88-417F-44C0-9345-DCEDDB5C41F8", + "categoryId": "C7324EF3-D951-45D9-A345-A82EAE344394", + "sku": "SH-M897-S", + "name": "Men's Sports Shorts, S", + "description": "The product called \"Men's Sports Shorts, S\"", + "price": 59.99 + }, + { + "id": "F7261436-B748-42D6-A7C9-ACD2B589F0B7", + "categoryId": "32A9A8E6-7004-4B24-9C2A-BB3E93B9E6BD", + "sku": "GL-H102-S", + "name": "Half-Finger Gloves, S", + "description": "The product called \"Half-Finger Gloves, S\"", + "price": 24.49 + }, + { + "id": "FDD4E68A-6284-4DC7-B48D-232F347CA827", + "categoryId": "3B75F01D-6443-4C83-B182-8BB38192C33B", + "sku": "FR-M94B-44", + "name": "HL Mountain Frame - Black, 44", + "description": "The product called \"HL Mountain Frame - Black, 44\"", + "price": 1349.6 + }, + { + "id": "FE292D83-1F34-4845-A467-7C62AD3C6CBE", + "categoryId": "7FF64215-1F7A-4CDF-9BA1-AD6ADC6B5D1C", + "sku": "PU-M044", + "name": "Mountain Pump", + "description": "The product called \"Mountain Pump\"", + "price": 24.99 + }, + { + "id": "FFCA3096-199F-41C3-99D1-35BC88D8AC6F", + "categoryId": "3E4CEACD-D007-46EB-82D7-31F6141752B2", + "sku": "FR-R72Y-48", + "name": "ML Road Frame-W - Yellow, 48", + "description": "The product called \"ML Road Frame-W - Yellow, 48\"", + "price": 594.83 + } +] diff --git a/Labs/vector_search/.env.example b/Labs/lab_3_vector_search/.env.example similarity index 100% rename from Labs/vector_search/.env.example rename to Labs/lab_3_vector_search/.env.example diff --git a/Labs/vector_search/README.md b/Labs/lab_3_vector_search/README.md similarity index 67% rename from Labs/vector_search/README.md rename to Labs/lab_3_vector_search/README.md index 06e3708..c39fe29 100644 --- a/Labs/vector_search/README.md +++ b/Labs/lab_3_vector_search/README.md @@ -1,14 +1,18 @@ # Lab - Vector search using vCore-based Azure Cosmos DB for MongoDB -This lab demonstrates using an Azure OpenAI embedding model to vectorize documents already stored in Azure Cosmos DB API for MongoDB, storing the embedding vectors and the creation of a vector index. The lab also demonstrates how to use the vector index to perform vector searches, and how to use the vector search results in a RAG (Retrieval Augmented Generation) pattern to generate a response from a large language model (LLM). +This lab demonstrates using an Azure OpenAI embedding model to vectorize documents already stored in Azure Cosmos DB API for MongoDB, storing the embedding vectors and the creation of a vector index. +The lab also demonstrates how to use the vector index to perform vector searches, and how to use the vector search results in a RAG (Retrieval Augmented Generation) pattern to generate a response from a large language model (LLM). ## Pre-requisites -This lab expects that data is already loaded into the Azure Cosmos DB API for MongoDB collections. If you do not have data, you can run the `catch_up.js` script to load sample data into the collections. +This lab expects that data is already loaded into the Azure Cosmos DB API for MongoDB collections. +If you do not have data, you can run the `catch_up.js` script to load sample data into the collections. -1. In the Visual Studio Code terminal, ensure you are in the `vector_search` lab directory. +1. +In the Visual Studio Code terminal, ensure you are in the `vector_search` lab directory. -2. Run the `catch_up.js` script to load sample data into the collections. +2. +Run the `catch_up.js` script to load sample data into the collections. ```bash node catch_up.js @@ -16,9 +20,11 @@ This lab expects that data is already loaded into the Azure Cosmos DB API for Mo ## Setup the lab environment -1. Open the `.env` file in the Visual Studio Code editor. +1. +Open the `.env` file in the Visual Studio Code editor. -2. Add the following settings to the `.env` file, replacing the values from the deployed Azure OpenAI service: +2. +Add the following settings to the `.env` file, replacing the values from the deployed Azure OpenAI service: ```bash AOAI_ENDPOINT=https://-openai.openai.azure.com/ @@ -27,9 +33,11 @@ This lab expects that data is already loaded into the Azure Cosmos DB API for Mo Replace `` with the name of the deployed OpenAI service, and `` with the Azure OpenAI API key. -3. In Visual Studio Code, open a terminal window and navigate to the lab folder `vector_search`. +3. +In Visual Studio Code, open a terminal window and navigate to the lab folder `vector_search`. -4. Install the required packages by running the following command in the terminal window: +4. +Install the required packages by running the following command in the terminal window: ```bash npm install @@ -37,15 +45,18 @@ This lab expects that data is already loaded into the Azure Cosmos DB API for Mo ## Create the Azure OpenAI client -1. Open the `index.js` file in the Visual Studio Code editor. +1. +Open the `index.js` file in the Visual Studio Code editor. -2. Beneath the line: `const { MongoClient } = require('mongodb');`, add the following code to import the Azure OpenAI client and Azure Key Credential classes: +2. +Beneath the line: `const { MongoClient } = require('mongodb');`, add the following code to import the Azure OpenAI client and Azure Key Credential classes: ```javascript const { OpenAIClient, AzureKeyCredential} = require("@azure/openai"); ``` -3. Beneath the code that sets up the MongoDB client, add the following code to create an instance of the Azure OpenAI client: +3. +Beneath the code that sets up the MongoDB client, add the following code to create an instance of the Azure OpenAI client: ```javascript // set up the Azure OpenAI client @@ -57,9 +68,12 @@ This lab expects that data is already loaded into the Azure Cosmos DB API for Mo ## Create a function to generate text embeddings -Vectorizing or embedding text is the process of converting text into a numerical representation. This guide has deployed an embedding model to Azure OpenAI, which can be used to generate embeddings for text. In this section, a function is introduced that uses the Azure OpenAI model to generate embeddings for text. +Vectorizing or embedding text is the process of converting text into a numerical representation. +This guide has deployed an embedding model to Azure OpenAI, which can be used to generate embeddings for text. +In this section, a function is introduced that uses the Azure OpenAI model to generate embeddings for text. -1. In `index.js`, add the following code directly above the last line of the file (that calls the `main` function): +1. +In `index.js`, add the following code directly above the last line of the file (that calls the `main` function): ```javascript async function generateEmbeddings(text) { @@ -70,15 +84,18 @@ Vectorizing or embedding text is the process of converting text into a numerical } ``` -2. In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to test the new `generateEmbeddings` function: +2. +In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to test the new `generateEmbeddings` function: ```javascript console.log(await generateEmbeddings("Hello, world!")); ``` -3. Save the `index.js` file. +3. +Save the `index.js` file. -4. Run the application and observe the vector output in the console. +4. +Run the application and observe the vector output in the console. ```bash npm start @@ -86,19 +103,26 @@ Vectorizing or embedding text is the process of converting text into a numerical ![Console output displays a large numerical vector.](media/text_embedding_output.png "Vector representation of text") -5. Remove the line of code from step 2. +5. +Remove the line of code from step 2. -6. Save the file. +6. +Save the file. ## Vectorize and store the embeddings in each document -Now that the `generateEmbeddings` function is working, the next step is to use it to generate embeddings for each document and store the embeddings in a new field (contentVector) within the same document. The process of creating a vector embedding field on each document only needs to be done once. However, if a document changes, the vector embedding field will need to be updated with an updated vector. +Now that the `generateEmbeddings` function is working, the next step is to use it to generate embeddings for each document and store the embeddings in a new field (contentVector) within the same document. +The process of creating a vector embedding field on each document only needs to be done once. +However, if a document changes, the vector embedding field will need to be updated with an updated vector. -In this section, a function is added that will loop through each document in a collection, generate the vector embedding, and store the vector embedding in the document itself. The function takes advantage of `bulkWrite` operations to perform upserts on the existing documents in an efficient way. Lastly, the function will create a vector index (VectorSearchIndex) on the collection to enable vector search queries if it does not already exist. +In this section, a function is added that will loop through each document in a collection, generate the vector embedding, and store the vector embedding in the document itself. +The function takes advantage of `bulkWrite` operations to perform upserts on the existing documents in an efficient way. +Lastly, the function will create a vector index (VectorSearchIndex) on the collection to enable vector search queries if it does not already exist. >**Note**: Generating embedding vectors for each document will take some time, exercise patience during this section until the process is complete. -1. In `index.js`, add the following code directly above the last line of the file (that calls the `main` function) - the code is documented inline to explain the steps taken: +1. +In `index.js`, add the following code directly above the last line of the file (that calls the `main` function) - the code is documented inline to explain the steps taken: ```javascript async function addCollectionContentVectorField(db, collectionName) { @@ -161,15 +185,18 @@ In this section, a function is added that will loop through each document in a c } ``` -2. In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to call the new `addCollectionContentVectorField` function on the `products` collection: +2. +In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to call the new `addCollectionContentVectorField` function on the `products` collection: ```javascript await addCollectionContentVectorField(db, 'products'); ``` -3. Save the `index.js` file. +3. +Save the `index.js` file. -4. Run the application and observe the progress in the console. +4. +Run the application and observe the progress in the console. ```bash npm start @@ -177,17 +204,23 @@ In this section, a function is added that will loop through each document in a c ![Console output displays the progress of vectorizing and storing the embeddings in each document.](media/vectorize_and_store_embeddings.png "Vectorizing and storing the embeddings in each document") -5. Repeat steps 2-4 for the `customers` and `sales` collections by modifying the collection name in the `addCollectionContentVectorField` function call (each collection only needs to be vectorized once). +5. +Repeat steps 2-4 for the `customers` and `sales` collections by modifying the collection name in the `addCollectionContentVectorField` function call (each collection only needs to be vectorized once). -6. Once complete, remove all calls to the `addCollectionContentVectorField` function from the main function. +6. +Once complete, remove all calls to the `addCollectionContentVectorField` function from the main function. -7. Save the file. +7. +Save the file. ## Use vector search in vCore-based Azure Cosmos DB for MongoDB -Now that each document has its associated vector embedding and the vector indexes have been created on each collection, we can now use the vector search capabilities of vCore-based Azure Cosmos DB for MongoDB. In this section, a function is added that will perform a vector search query that will return the most relevant documents based on the cosine similarity of the query vector and the content vectors of the documents in the collection. +Now that each document has its associated vector embedding and the vector indexes have been created on each collection, we can now use the vector search capabilities of vCore-based Azure Cosmos DB for MongoDB. +In this section, a function is added that will perform a vector search query that will return the most relevant documents based on the cosine similarity of the query vector and the content vectors of the documents in the collection. -1. In `index.js`, add the following code directly above the last line of the file (that calls the `main` function) - the code is documented inline to explain the steps taken. This code introduces two functions, one to perform a vector search and another to neatly print the search results: +1. +In `index.js`, add the following code directly above the last line of the file (that calls the `main` function) - the code is documented inline to explain the steps taken. +This code introduces two functions, one to perform a vector search and another to neatly print the search results: ```javascript async function vectorSearch(db, collectionName, query, numResults = 3) { @@ -224,7 +257,8 @@ Now that each document has its associated vector embedding and the vector indexe } ``` -2. In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to test the new `vectorSearch` function: +2. +In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to test the new `vectorSearch` function: ```javascript //vector search for the top 3 most relevant products @@ -232,9 +266,11 @@ Now that each document has its associated vector embedding and the vector indexe searchResults.forEach(printProductSearchResult); ``` -3. Save the `index.js` file. +3. +Save the `index.js` file. -4. Run the application and observe the search results in the console. +4. +Run the application and observe the search results in the console. ```bash npm start @@ -242,15 +278,20 @@ Now that each document has its associated vector embedding and the vector indexe ![Console output displays the search results.](media/vector_search_results.png "Vector search results") -5. Delete the code added in step 2. +5. +Delete the code added in step 2. -6. Save the file. +6. +Save the file. ## Use vector search in a RAG (Retrieval Augmented Generation) pattern -In this section, a function is added that will use the vector search results to augment a prompt to a large language model (LLM). This is considered a RAG (Retrieval Augmented Generation) pattern. The function will use the vector search results to retrieve the most relevant documents and then use the unvectorized text of the retrieved documents to in the prompt to get an accurate response from the LLM. +In this section, a function is added that will use the vector search results to augment a prompt to a large language model (LLM). +This is considered a RAG (Retrieval Augmented Generation) pattern. +The function will use the vector search results to retrieve the most relevant documents and then use the unvectorized text of the retrieved documents to in the prompt to get an accurate response from the LLM. -1. In `index.js`, add the following code directly above the last line of the file (that calls the `main` function) - the code is documented inline to explain the steps taken: +1. +In `index.js`, add the following code directly above the last line of the file (that calls the `main` function) - the code is documented inline to explain the steps taken: ```javascript async function ragWithVectorsearch(db, collectionName, question, numResults=3) { @@ -299,16 +340,19 @@ In this section, a function is added that will use the vector search results to } ``` -2. In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to test the new `ragWithVectorsearch` function: +2. +In the `main` function, beneath the `const db = dbClient.db('cosmic_works');` line of code, add the following code to test the new `ragWithVectorsearch` function: ```javascript //RAG with vector search for the top 3 most relevant products console.log(await ragWithVectorsearch(db, 'products', 'What are the names and skus of some of the bikes you have?', 3)); ``` -3. Save the `index.js` file. +3. +Save the `index.js` file. -4. Run the application and observe the response in the console. +4. +Run the application and observe the response in the console. ```bash npm start @@ -318,4 +362,5 @@ In this section, a function is added that will use the vector search results to ## Summary -In this lab, you have learned how to use an Azure OpenAI embedding model to vectorize documents already stored in Azure Cosmos DB API for MongoDB, store the embedding vectors, and create a vector index. You have also learned how to use the vector index to perform vector searches, and how to use the vector search results in a RAG (Retrieval Augmented Generation) pattern to generate a response from a large language model (LLM). +In this lab, you have learned how to use an Azure OpenAI embedding model to vectorize documents already stored in Azure Cosmos DB API for MongoDB, store the embedding vectors, and create a vector index. +You have also learned how to use the vector index to perform vector searches, and how to use the vector search results in a RAG (Retrieval Augmented Generation) pattern to generate a response from a large language model (LLM). diff --git a/Labs/vector_search/catch_up.js b/Labs/lab_3_vector_search/catch_up.js similarity index 100% rename from Labs/vector_search/catch_up.js rename to Labs/lab_3_vector_search/catch_up.js diff --git a/Labs/vector_search/index.js b/Labs/lab_3_vector_search/index.js similarity index 100% rename from Labs/vector_search/index.js rename to Labs/lab_3_vector_search/index.js diff --git a/Labs/vector_search/media/rag_with_vector_search_response.png b/Labs/lab_3_vector_search/media/rag_with_vector_search_response.png similarity index 100% rename from Labs/vector_search/media/rag_with_vector_search_response.png rename to Labs/lab_3_vector_search/media/rag_with_vector_search_response.png diff --git a/Labs/vector_search/media/text_embedding_output.png b/Labs/lab_3_vector_search/media/text_embedding_output.png similarity index 100% rename from Labs/vector_search/media/text_embedding_output.png rename to Labs/lab_3_vector_search/media/text_embedding_output.png diff --git a/Labs/vector_search/media/vector_search_results.png b/Labs/lab_3_vector_search/media/vector_search_results.png similarity index 100% rename from Labs/vector_search/media/vector_search_results.png rename to Labs/lab_3_vector_search/media/vector_search_results.png diff --git a/Labs/vector_search/media/vectorize_and_store_embeddings.png b/Labs/lab_3_vector_search/media/vectorize_and_store_embeddings.png similarity index 100% rename from Labs/vector_search/media/vectorize_and_store_embeddings.png rename to Labs/lab_3_vector_search/media/vectorize_and_store_embeddings.png diff --git a/Labs/vector_search/package.json b/Labs/lab_3_vector_search/package.json similarity index 100% rename from Labs/vector_search/package.json rename to Labs/lab_3_vector_search/package.json diff --git a/Labs/langchain/.env.example b/Labs/lab_4_langchain/.env.example similarity index 100% rename from Labs/langchain/.env.example rename to Labs/lab_4_langchain/.env.example diff --git a/Labs/langchain/README.md b/Labs/lab_4_langchain/README.md similarity index 64% rename from Labs/langchain/README.md rename to Labs/lab_4_langchain/README.md index 9913f7d..f682d71 100644 --- a/Labs/langchain/README.md +++ b/Labs/lab_4_langchain/README.md @@ -1,6 +1,8 @@ # Lab - LangChain -In the previous lab, the `mongodb` package was used to perform a vector search through a db command to find product documents that were most similar to the user's input. In this lab, you will use the LangChain package to perform the same search. LangChain has a vector store class named `AzureCosmosDBVectorStore`, a community contribution, that supports vector search in vCore-based Azure Cosmos DB for MongoDB. +In the previous lab, the `mongodb` package was used to perform a vector search through a db command to find product documents that were most similar to the user's input. +In this lab, you will use the LangChain package to perform the same search. +LangChain has a vector store class named `AzureCosmosDBVectorStore`, a community contribution, that supports vector search in vCore-based Azure Cosmos DB for MongoDB. The `AzureCosmosDBVectorStore` class represents a single vector index in the database, therefore for the instructions for this lab will focus on the `products` collection, however the same steps can be used with the `customers` and `sales` collections as well. @@ -8,11 +10,14 @@ Throughout this lab, notice how much more concise the code is compared to the pr ## Setup the lab environment -To simplify code, the LangChain package makes use of environment variables. The `.env` file in this lab will reflect the naming conventions of the LangChain packages and differs from previous labs. +To simplify code, the LangChain package makes use of environment variables. +The `.env` file in this lab will reflect the naming conventions of the LangChain packages and differs from previous labs. -1. Open the `.env` file in the Visual Studio Code editor. +1. +Open the `.env` file in the Visual Studio Code editor. -2. Add the following settings to the `.env` file, populating the MongoDB connection string and replacing the values from the deployed Azure OpenAI service: +2. +Add the following settings to the `.env` file, populating the MongoDB connection string and replacing the values from the deployed Azure OpenAI service: ```bash AZURE_COSMOSDB_CONNECTION_STRING= @@ -27,9 +32,11 @@ To simplify code, the LangChain package makes use of environment variables. The >**Note**: The Azure OpenAI service name is not the full endpoint. Only the service name is required. For example, if the endpoint is `https://myservicename.openai.azure.com/`, then the service name is `myservicename`. -3. In Visual Studio Code, open a terminal window and navigate to the lab folder `langchain`. +3. +In Visual Studio Code, open a terminal window and navigate to the lab folder `langchain`. -4. Install the required packages by running the following command in the terminal window: +4. +Install the required packages by running the following command in the terminal window: ```bash npm install @@ -37,13 +44,18 @@ To simplify code, the LangChain package makes use of environment variables. The ## Pre-requisites -This lab expects that data is already loaded into the Azure Cosmos DB API for MongoDB collections, and the `contentVector` field is populated (along with a vector index created). You can run the `catch_up.js` script to load sample data, generate/populate the `contentVector` field and create the vector search index in each collection. +This lab expects that data is already loaded into the Azure Cosmos DB API for MongoDB collections, and the `contentVector` field is populated (along with a vector index created). +You can run the `catch_up.js` script to load sample data, generate/populate the `contentVector` field and create the vector search index in each collection. ->**Note**: This script will take a few minutes to run. This script will load data, generate the `contentVector` field, and create the vector index in the `products` collection only. This `products` collection is the focus of the instructions provided in this lab. +>**Note**: This script will take a few minutes to run. +This script will load data, generate the `contentVector` field, and create the vector index in the `products` collection only. +This `products` collection is the focus of the instructions provided in this lab. -1. In the Visual Studio Code terminal, ensure you are in the `langchain` lab directory. +1. +In the Visual Studio Code terminal, ensure you are in the `langchain` lab directory. -2. Run the `catch_up.js` script to satisfy the lab pre-requisites. +2. +Run the `catch_up.js` script to satisfy the lab pre-requisites. ```bash node catch_up.js @@ -51,15 +63,24 @@ This lab expects that data is already loaded into the Azure Cosmos DB API for Mo ## Vector search with LangChain -The first step is to initialize the connection to the vector store. The `AzureCosmosDBVectorStore` class is used to represent the vector store. This section will walk through the initialization of this connection as well as a test call that issues a vector search and outputs the results. +The first step is to initialize the connection to the vector store. +The `AzureCosmosDBVectorStore` class is used to represent the vector store. +This section will walk through the initialization of this connection as well as a test call that issues a vector search and outputs the results. -When establishing the connection to the vector store (vCore-based Azure Cosmos DB for MongoDB), recall that in the previous lab, each collection was populated and a `contentVector` field added that contains the vectorized embeddings of the document itself. Finally, a vector index was also created on the `contentVector` field to enable vector search. The vector index in each collection is named `VectorSearchIndex`. +When establishing the connection to the vector store (vCore-based Azure Cosmos DB for MongoDB), recall that in the previous lab, each collection was populated and a `contentVector` field added that contains the vectorized embeddings of the document itself. +Finally, a vector index was also created on the `contentVector` field to enable vector search. +The vector index in each collection is named `VectorSearchIndex`. -The return value of a vector search in LangChain is a list of `Document` objects. The LangChain `Document` class contains two properties: `pageContent`, that represents the textual content that is typically used to augment the prompt, and `metadata` that contains all other attributes of the document. In the cell below, we'll use the `_id` field as the `pageContent`, and the rest of the fields are returned as metadata. +The return value of a vector search in LangChain is a list of `Document` objects. +The LangChain `Document` class contains two properties: `pageContent`, that represents the textual content that is typically used to augment the prompt, and `metadata` that contains all other attributes of the document. +In the cell below, we'll use the `_id` field as the `pageContent`, and the rest of the fields are returned as metadata. -1. Open the `index.js` file in the Visual Studio Code editor. +1. +Open the `index.js` file in the Visual Studio Code editor. -2. Directly beneath the line `const { MongoClient } = require('mongodb');`, add the following code to import the necessary LangChain packages. This code imports the AzureCosmosDBVectorStore that represents the vector index in a collection, the `AzureCosmosDBSimilarityType` that will allow us to perform a vector search using cosine similarity, and the `OpenAIEmbeddings` class that will be used to generate the embeddings for the user's input for the vector search. +2. +Directly beneath the line `const { MongoClient } = require('mongodb');`, add the following code to import the necessary LangChain packages. +This code imports the AzureCosmosDBVectorStore that represents the vector index in a collection, the `AzureCosmosDBSimilarityType` that will allow us to perform a vector search using cosine similarity, and the `OpenAIEmbeddings` class that will be used to generate the embeddings for the user's input for the vector search. ```javascript const { AzureCosmosDBVectorStore, @@ -68,7 +89,9 @@ The return value of a vector search in LangChain is a list of `Document` objects const { OpenAIEmbeddings } = require("@langchain/openai") ``` -3. Directly beneath the line of code that sets up the MongoDB connection `const dbClient = new MongoClient(process.env.AZURE_COSMOSDB_CONNECTION_STRING);`, add the following code to initialize the connection to the vector store that points to the `products` collection and associated index. The `OpenAIEmbeddings` uses the default values obtained from the environment variables to initialize. +3. +Directly beneath the line of code that sets up the MongoDB connection `const dbClient = new MongoClient(process.env.AZURE_COSMOSDB_CONNECTION_STRING);`, add the following code to initialize the connection to the vector store that points to the `products` collection and associated index. +The `OpenAIEmbeddings` uses the default values obtained from the environment variables to initialize. ```javascript // set up the Azure Cosmos DB vector store using the initialized MongoDB client @@ -83,7 +106,8 @@ The return value of a vector search in LangChain is a list of `Document` objects const vectorStore = new AzureCosmosDBVectorStore(new OpenAIEmbeddings(), azureCosmosDBConfig); ``` -4. In the `main` function, directly beneath the line of code `console.log("Connected to MongoDB");`, append the following code that will perform a vector search using the `vectorStore` object and output the results. +4. +In the `main` function, directly beneath the line of code `console.log("Connected to MongoDB");`, append the following code that will perform a vector search using the `vectorStore` object and output the results. ```javascript // perform a vector search using the vector store @@ -95,9 +119,11 @@ The return value of a vector search in LangChain is a list of `Document` objects console.log(results); ``` -5. Save the `index.js` file. +5. +Save the `index.js` file. -6. Run the application by executing the following command in the terminal window: +6. +Run the application by executing the following command in the terminal window: ```bash npm start @@ -109,19 +135,29 @@ The return value of a vector search in LangChain is a list of `Document` objects ![A portion of the console output shows search results with products that are similar to the user query.](media/initial_vector_search.png "Initial vector search results") -7. Delete or comment out the code added in step 4. +7. +Delete or comment out the code added in step 4. -8. Save the `index.js` file. +8. +Save the `index.js` file. ## RAG with LangChain -In this section, we'll implement the RAG pattern using LangChain. In LangChain, a **retriever** is used to augment the prompt with contextual data. In this case, the already established vector store will be used as the retriever. By default, the prompt is augmented with the `pageContent` field of the retrieved document that customarily contains the text content of the embedded vector. In our case, the document itself serves as the textual content, so we'll have to do some pre-processing to format the text of the product list that is expected in our system prompt (JSON string) - see the `formatDocuments` function below for this implementation. +In this section, we'll implement the RAG pattern using LangChain. +In LangChain, a **retriever** is used to augment the prompt with contextual data. +In this case, the already established vector store will be used as the retriever. +By default, the prompt is augmented with the `pageContent` field of the retrieved document that customarily contains the text content of the embedded vector. +In our case, the document itself serves as the textual content, so we'll have to do some pre-processing to format the text of the product list that is expected in our system prompt (JSON string) - see the `formatDocuments` function below for this implementation. -We'll also define a reusable RAG chain to control the flow and behavior of the call into the LLM. This chain is defined using the LCEL syntax (LangChain Expression Language). +We'll also define a reusable RAG chain to control the flow and behavior of the call into the LLM. +This chain is defined using the LCEL syntax (LangChain Expression Language). -1. Open the `index.js` file in the Visual Studio Code editor. +1. +Open the `index.js` file in the Visual Studio Code editor. -2. Directly preceding the final line of code in this file (where the main function is called), add the `formatDocuments` function that will be used to format the `Document` objects into a JSON string that will be used in the prompt for the LLM. The code is documented inline. +2. +Directly preceding the final line of code in this file (where the main function is called), add the `formatDocuments` function that will be used to format the `Document` objects into a JSON string that will be used in the prompt for the LLM. +The code is documented inline. ```javascript function formatDocuments(docs) { @@ -154,13 +190,15 @@ We'll also define a reusable RAG chain to control the flow and behavior of the c } ``` -3. In the `require("@langchain/openai")` import statement, add the `ChatOpenAI` class that will be used to interact with the LLM. +3. +In the `require("@langchain/openai")` import statement, add the `ChatOpenAI` class that will be used to interact with the LLM. ```javascript const { OpenAIEmbeddings, ChatOpenAI } = require("@langchain/openai") ``` -4. Directly beneath the line of code that was just modified in the previous step, add the following: +4. +Directly beneath the line of code that was just modified in the previous step, add the following: ```javascript // To support the LangChain LCEL RAG chain @@ -169,14 +207,17 @@ We'll also define a reusable RAG chain to control the flow and behavior of the c const { StringOutputParser } = require("@langchain/core/output_parsers") ``` -5. Immediately before the `async function main()` function, add the following code to initialize an Azure OpenAI chat model. +5. +Immediately before the `async function main()` function, add the following code to initialize an Azure OpenAI chat model. ```javascript // set up the OpenAI chat model const chatModel = new ChatOpenAI(); ``` -6. Before the last line of code in the file (that calls the main function), add the following function that creates a reusable LangChain RAG chain. The code is documented inline. +6. +Before the last line of code in the file (that calls the main function), add the following function that creates a reusable LangChain RAG chain. +The code is documented inline. ```javascript async function ragLCELChain(question) { @@ -228,15 +269,18 @@ We'll also define a reusable RAG chain to control the flow and behavior of the c } ``` -7. In the `main` function, directly beneath the line of code `console.log('Connected to MongoDB');`, add the following code that will call the `ragLCELChain` function to perform a vector search using the `vectorStore` object and output the results. +7. +In the `main` function, directly beneath the line of code `console.log('Connected to MongoDB');`, add the following code that will call the `ragLCELChain` function to perform a vector search using the `vectorStore` object and output the results. ```javascript console.log(await ragLCELChain("What yellow products do you have?")); ``` -8. Save the `index.js` file. +8. +Save the `index.js` file. -9. Run the application by executing the following command in the terminal window: +9. +Run the application by executing the following command in the terminal window: ```bash npm start @@ -244,17 +288,27 @@ We'll also define a reusable RAG chain to control the flow and behavior of the c ![The console output shows the response from the LLM based on the augmented prompt and returns the LLM response.](media/rag_chain_output.png "RAG chain output") -10. Delete or comment out the code added in step 7. +10. +Delete or comment out the code added in step 7. -11. Save the `index.js` file. +11. +Save the `index.js` file. ## LangChain agent -In this section, we'll implement a LangChain agent that will be used to interact with the LLM. The agent will be used to control the flow and behavior of the call into the LLM. An agent differs from the previous RAG chain in that the RAG chain is a definition of a sequence of specific actions. Agents on the other hand are equipped with various tools and will determine which actions to take based on the context of the conversation. In this scenario, an agent will be equipped with two tools, one that uses a retriever chain from the vector store, and another that will perform a MongoDB lookup based on a SKU value. The reason the SKU lookup tool is introduced is because the vector store is built for semantic search. Some fields, such as the `_id` or `sku` fields do not have semantic meaning and therefore a direct lookup is more appropriate. +In this section, we'll implement a LangChain agent that will be used to interact with the LLM. +The agent will be used to control the flow and behavior of the call into the LLM. +An agent differs from the previous RAG chain in that the RAG chain is a definition of a sequence of specific actions. +Agents on the other hand are equipped with various tools and will determine which actions to take based on the context of the conversation. +In this scenario, an agent will be equipped with two tools, one that uses a retriever chain from the vector store, and another that will perform a MongoDB lookup based on a SKU value. +The reason the SKU lookup tool is introduced is because the vector store is built for semantic search. +Some fields, such as the `_id` or `sku` fields do not have semantic meaning and therefore a direct lookup is more appropriate. -1. Open the `index.js` file in the Visual Studio Code editor. +1. +Open the `index.js` file in the Visual Studio Code editor. -2. Immediately after the `const { StringOutputParser } = require("@langchain/core/output_parsers");` line of code, add the following dependencies to support the creation of the LangChain agent: +2. +Immediately after the `const { StringOutputParser } = require("@langchain/core/output_parsers");` line of code, add the following dependencies to support the creation of the LangChain agent: ```javascript // For LangChain agent @@ -266,7 +320,9 @@ In this section, we'll implement a LangChain agent that will be used to interact const { formatToOpenAIFunctionMessages } = require("langchain/agents/format_scratchpad"); ``` -3. Immediately before the last line of code in the file (that calls the main function), add the following function that builds a LangChain agent executor and a helper function that will invoke the agent. The code is documented inline. +3. +Immediately before the last line of code in the file (that calls the main function), add the following function that builds a LangChain agent executor and a helper function that will invoke the agent. +The code is documented inline. ```javascript async function buildAgentExecutor() { @@ -376,16 +432,19 @@ In this section, we'll implement a LangChain agent that will be used to interact } ``` -4. In the `main` function, directly beneath the line of code `console.log('Connected to MongoDB');`, add the following code that will call the `buildAgentExecutor` function to create the LangChain agent executor and the `executeAgent` function to engage with the agent and outupt the results. +4. +In the `main` function, directly beneath the line of code `console.log('Connected to MongoDB');`, add the following code that will call the `buildAgentExecutor` function to create the LangChain agent executor and the `executeAgent` function to engage with the agent and outupt the results. ```javascript const agentExecutor = await buildAgentExecutor(); console.log(await executeAgent(agentExecutor, "What yellow products do you have?")); ``` -5. Save the `index.js` file. +5. +Save the `index.js` file. -6. Run the application by executing the following command in the terminal window: +6. +Run the application by executing the following command in the terminal window: ```bash npm start @@ -393,13 +452,18 @@ In this section, we'll implement a LangChain agent that will be used to interact ![The console output shows the response from the LangChain agent based on the user input.](media/agent_output.png "LangChain agent output") -7. Change the question in the `executeAgent` function to `What is the name of the product that has the SKU TI-R982?`. +7. +Change the question in the `executeAgent` function to `What is the name of the product that has the SKU TI-R982?`. -8. Find the commented line of code `//returnIntermediateSteps: true` in the `executeAgent` function and uncomment it to enable verbose output of the tool usage of the agent. This will output the intermediate steps of the agent that includes the function calls and their outputs. +8. +Find the commented line of code `//returnIntermediateSteps: true` in the `executeAgent` function and uncomment it to enable verbose output of the tool usage of the agent. +This will output the intermediate steps of the agent that includes the function calls and their outputs. -9. Save the `index.js` file. +9. +Save the `index.js` file. -10. Run the application by executing the following command in the terminal window: +10. +Run the application by executing the following command in the terminal window: ```bash npm start @@ -409,10 +473,14 @@ In this section, we'll implement a LangChain agent that will be used to interact Notice the verbose output includes the selection, input, and observation(output) of the tool used by the agent. -11. If desired, comment out the line of code `//returnIntermediateSteps: true` in the `executeAgent` function to disable verbose output of the tool usage of the agent. +11. +If desired, comment out the line of code `//returnIntermediateSteps: true` in the `executeAgent` function to disable verbose output of the tool usage of the agent. -12. Experiment with additional questions of your own. +12. +Experiment with additional questions of your own. ## Summary -In this lab, you used the LangChain package to perform a vector search in vCore-based Azure Cosmos DB for MongoDB. You initialized the connection to the vector store, performed a vector search, implemented the RAG pattern using LangChain, and created a LangChain agent. You also experimented with the LangChain agent and observed the verbose output of the tool usage of the agent. +In this lab, you used the LangChain package to perform a vector search in vCore-based Azure Cosmos DB for MongoDB. +You initialized the connection to the vector store, performed a vector search, implemented the RAG pattern using LangChain, and created a LangChain agent. +You also experimented with the LangChain agent and observed the verbose output of the tool usage of the agent. diff --git a/Labs/langchain/catch_up.js b/Labs/lab_4_langchain/catch_up.js similarity index 100% rename from Labs/langchain/catch_up.js rename to Labs/lab_4_langchain/catch_up.js diff --git a/Labs/langchain/index.js b/Labs/lab_4_langchain/index.js similarity index 100% rename from Labs/langchain/index.js rename to Labs/lab_4_langchain/index.js diff --git a/Labs/langchain/media/agent_output.png b/Labs/lab_4_langchain/media/agent_output.png similarity index 100% rename from Labs/langchain/media/agent_output.png rename to Labs/lab_4_langchain/media/agent_output.png diff --git a/Labs/langchain/media/agent_verbose_output.png b/Labs/lab_4_langchain/media/agent_verbose_output.png similarity index 100% rename from Labs/langchain/media/agent_verbose_output.png rename to Labs/lab_4_langchain/media/agent_verbose_output.png diff --git a/Labs/langchain/media/initial_vector_search.png b/Labs/lab_4_langchain/media/initial_vector_search.png similarity index 100% rename from Labs/langchain/media/initial_vector_search.png rename to Labs/lab_4_langchain/media/initial_vector_search.png diff --git a/Labs/langchain/media/rag_chain_output.png b/Labs/lab_4_langchain/media/rag_chain_output.png similarity index 100% rename from Labs/langchain/media/rag_chain_output.png rename to Labs/lab_4_langchain/media/rag_chain_output.png diff --git a/Labs/langchain/package.json b/Labs/lab_4_langchain/package.json similarity index 100% rename from Labs/langchain/package.json rename to Labs/lab_4_langchain/package.json diff --git a/Labs/lab_5_backend_api/README.md b/Labs/lab_5_backend_api/README.md new file mode 100644 index 0000000..a60f155 --- /dev/null +++ b/Labs/lab_5_backend_api/README.md @@ -0,0 +1,280 @@ +# Lab 5 - Backend API + +In the previous lab, a LangChain agent was created armed with tools to do vector lookups and concrete document id lookups via function calling. +In this lab, the agent functionality needs to be extracted into a backend api for the frontend application that will allow users to interact with the agent. + +The information provided in this section assumes that the dependent infrastructure is deployed and have completed the previous labs in this dev guide. + +## Overview + +The backend api is a Node.js web application using Express and Swagger that will expose endpoints for the frontend application to interact with. +The backend api is a containerized application that will be deployed to [Azure Container Apps](https://learn.microsoft.com/azure/container-apps/overview). + +## Clone the Backend API + +Create a folder to house the repository. +Open a terminal and navigate to the folder. +Clone the repository, then navigate to the `Backend` folder within the repository. + +```bash +git clone https://github.com/AzureCosmosDB/Azure-OpenAI-Node.js-Developer-Guide.git + +cd Azure-OpenAI-Node.js-Developer-Guide +cd Backend +``` + +## Run the backend api locally + +When developing a backend api, it is often useful to run the application locally to test and debug. +This section outlines how to run the backend api locally while watching the file system for code changes. +Any detected changes will automatically restart the backend api. + +1. Open the backend api folder location in VS Code. + +2. Open a **Terminal** window in VS Code (CTRL+`). + +3. Setup the `.env` file. Copy the `.env.example` file to `.env` and update the values. + These are the same environment variables used in the previous labs. + + ```bash + cp .env.example .env + ``` + +4. Add the following settings to the `.env` file, populating the MongoDB connection string and replacing the values from the deployed Azure OpenAI service: + ```bash + AZURE_COSMOSDB_CONNECTION_STRING= + AZURE_OPENAI_API_INSTANCE_NAME= + AZURE_OPENAI_API_KEY= + AZURE_OPENAI_API_DEPLOYMENT_NAME=completions + AZURE_OPENAI_API_EMBEDDINGS_DEPLOYMENT_NAME=embeddings + AZURE_OPENAI_API_VERSION=2023-09-01-preview + ``` + + Replace `` with the MongoDB connection string. Replace `` with the name of the deployed OpenAI service, and `` with the Azure OpenAI API key. Leave all other values untouched. + + >**Note**: The Azure OpenAI service name is not the full endpoint. Only the service name is required. For example, if the endpoint is `https://myservicename.openai.azure.com/`, then the service name is `myservicename`. + +5. Run the following command to install the dependencies. + + ```bash + npm install + ``` + +6. Run the following command to start the backend api. + + ```bash + node --env-file=.env app.js + ``` + + ![The VSCode terminal window displays with the backend API started.](media/local_backend_running_console.png "Local backend api running") + +7. Open a browser and navigate to `http://localhost:4242/docs` to view the Swagger UI. + + ![The Swagger UI displays for the locally running backend api](media/local_backend_swagger_ui.png "Local backend api Swagger UI") + +8. Expand the **GET / Root** endpoint and select **Try it out**. + Select **Execute** to send the request. + The response should display a status of `ready`. + + ![The Swagger UI displays the GET / Root endpoint reponse that has a status of ready.](media/local_backend_swagger_ui_root_response.png "Local backend api Swagger UI Root response") + +9. Expand the **POST /ai** endpoint and select **Try it out**. + In the **Request body** field, enter the following JSON. + + ```json + { + "session_id": "abc123", + "prompt": "What was the price of the product with sku `FR-R92B-58`" + } + ``` + +10. Select **Execute** to send the request. + Observe that the response indicates the price as being `$1431.50`. + + ![The Swagger UI displays the POST /ai endpoint reponse that has a status of ready.](media/local_backend_swagger_ui_ai_response.png "Local backend api Swagger UI AI response") + +11. In the Terminal window, press CTRL+C to stop the backend api. + +## Build and run the backend api container locally in Docker Desktop + +When deployed to Azure, the backend api will be running in a container. +It is important to test the container locally to ensure it is working as expected. +Containers are important because they provide a consistent environment for the application to run in. +This consistency allows the application to run the same way in development, test, and production environments - whether they be locally or in the cloud. + +The backend api contains a `Dockerfile` that defines the container image and is used by Docker to build the container image. +The Dockerfile contains instructions for Docker to build the container image. +The container image is a snapshot of the application and its dependencies. +The container image can be thought of an installer for the application to be deployed as needed in any environment. + +The `Dockerfile` for the backend api is shown below. + +```dockerfile +FROM node:21 + +WORKDIR /usr/src/app +COPY package*.json ./ +RUN npm install +COPY . . + +EXPOSE 80 + +CMD [ "node", "--env-file=.env", "app.js", "port=80" ] +``` + +Notice the steps of installing the pip dependencies, and running the **uvicorn** command line similar to what was done in the previous section. + +1. Ensure Docker Desktop is running. + +2. Open a **Terminal** window in VS Code (CTRL+`). + +4. Run the following command to build the container image. + Once complete, a message displays the operation has `FINISHED`. + + ```bash + docker build --pull --rm -f "DOCKERFILE" -t devguidenodebackendapi:latest "." + ``` + + ![The VSCode terminal window displays the docker build command and the FINISHED message.](media/local_backend_docker_build.png "Local backend api Docker build") + +5. Lastly, run the container in Docker Desktop using the following command. + + ```bash + docker run -d -p 4242:80 --name devguide-backend-api devguidenodebackendapi:latest + ``` + + ![The VSCode terminal window displays the docker run command.](media/local_backend_docker_run.png "Local backend api Docker run") + +6. Open a browser and navigate to `http://localhost:4242/docs` to view the Swagger UI. + +7. Repeat steps 8-10 from the previous section to test the backend api running in a container on Docker Desktop. + +## Deploy the backend api to Azure Container Apps + +### Retrieve the Azure Container Registry login server and admin credentials + +The backend api container image needs to be pushed to an Azure Container Registry (ACR) before it can be deployed to Azure Container Apps. +The ACR is a private container registry that will store the container image. +Azure Container Apps will pull the container image from the ACR to deploy the backend api. + +1. In the Azure portal, open the provisioned resource group and locate and open the **Container Registry** resource. + +2. Expand the **Settings** section in the left-hand menu and select **Access keys** from the left-hand menu. + Record the **Login server** value and the **Username** and **Password** values for later use. + + ![The Azure portal displays the Container Registry resource with the Access keys menu item selected. +The login server, username and password values are highlighted.](media/acr_access_keys.png "Container Registry Access keys") + +### Push the backend api container image to the Azure Container Registry + +Earlier, the backend api container image was built locally. +Now that the ACR login server and admin credentials are known, the container image can be pushed to the Azure Container Registry. + +1. Return to the terminal window in VS Code. + +2. Run the following command to tag the container image with the ACR login server. + Replace the `` value. + This command will silently complete with no output. + + ```bash + docker tag devguidenodebackendapi:latest /devguidenodebackendapi:v1 + ``` + +3. Run the following command to log into the ACR. + Replace the ``, ``, and `` values. + The message `Login Succeeded` displays when the login is successful. + + ```bash + docker login -u -p + ``` + +4. Once authenticated, push the container image to the ACR using the following command. + Replace the `` value. + + ```bash + docker push /devguidenodebackendapi:v1 + ``` + + ![The VSCode terminal window displays the docker login and docker push commands.](media/local_backend_docker_push.png "Local backend api Docker push") + +### Deploy the backend api ACR container image to Azure Container Apps + +The last step is to deploy the backend api container image to Azure Container Apps. +Azure Container Apps is a fully managed serverless container platform that allows developers to deploy containerized applications without having to manage any infrastructure. + +1. In the Azure portal, open the provisioned resource group and locate and open the **Container App** resource, the name will end in `-api`. + Record the name of this resource. + +2. Back in the resource group, locate the **Container Apps Environment** resource, the name will end in `-containerappenv`. + Record the name of this resource. + +3. Also record the name of the resource group. + +4. Return to the terminal window in VS Code. + +5. Log into the Azure CLI using the following command. + + ```bash + az login + ``` + +6. Optionally set the current subscription to the correct subscription using the following command. + + ```bash + az account set --subscription + ``` + +7. Install the Azure Container Apps extension for the CLI using the following command. + + ```bash + az extension add --name containerapp --upgrade + ``` + +8. Run the following command to deploy the backend api container image to the existing Azure Container Apps resource. + Replace the ``, ``, ``, and `` values. + + ```bash + az containerapp up --name --image /devguidenodebackendapi:v1 --resource-group --environment --ingress external + ``` + ![The console output of the container app up command displays with the endpoint highlighted.](media/container_deploy.png) + +9. In the Azure Portal, locate and open the **Container App** resource ending in `-api`. + +10. From the left menu, expand the **Application** section and select **Revisions and replicas**. +Notice on the **Overview** screen, there is a failed container revision (may need to select **Refresh** from the top toolbar menu). + This is because the `hello-world` container is running at the same binding address as the backend api container. + + ![The Azure portal displays the Container App resource with a failed container revision listed.](media/container_app_failed_revision.png "Container App Failed Revision") + +11. View the error from the logs, or optionally expand the **Monitoring** section of the left menu and select **Log stream**. +Be sure to select the `-api` container. + + ![The Azure portal displays the Container App resource with the Log stream menu item selected and the output of the api container with the error highlighted.](media/container_app_log_stream.png "Container App Log Stream") + +12. To rectify this, the `hello-world` container needs to be deleted. +Expand the **Application** section of the left menu and select **Containers**. +Choose the **Edit and Deploy** button from the toolbar. + + ![The Azure portal displays the Container App resource with the Containers menu item selected. The Edit and Deploy button is highlighted.](media/container_app_edit_and_deploy.png "Container App Edit and Deploy") + +13. On the **Create and deploy new revision** screen, beneath the **Container image** heading, check the box next to the `hello-world` container, then select **Delete**. + + ![The Azure portal displays the Create and deploy new revision screen with the hello-world container selected.](media/container_app_delete_hello_world.png "Container App Delete hello-world") + +14. Select **Create** to deploy the new revision. + In less than a minute, the new revision is deployed. + +15. Refresh the browser, then from the left menu, select **Overview**. + Select the **Application URL** link. + + ![The Azure portal displays the Container App resource Overview screen with the Application URL highlighted.](media/container_app_overview.png "Container App Overview") + +16. The UI should show the status as `ready`. + + ![The Azure portal displays the Container App resource with the Overview menu item selected. The Application URL is highlighted.](media/container_app_ready.png "Container App Ready") + +17. In the address bar of the browser, append `/docs` to the URL and press ENTER to view the Swagger UI. + +18. Repeat steps 8-10 from the [Run the backend api locally section](#run-the-backend-api-locally) to test the backend api running in a container on Azure Container Apps. + +Congratulations on the successful deployment of the backend api to Azure Container Apps where it is ready to service the frontend application. diff --git a/Labs/backend_api/media/acr_access_keys.png b/Labs/lab_5_backend_api/media/acr_access_keys.png similarity index 100% rename from Labs/backend_api/media/acr_access_keys.png rename to Labs/lab_5_backend_api/media/acr_access_keys.png diff --git a/Labs/backend_api/media/container_app_delete_hello_world.png b/Labs/lab_5_backend_api/media/container_app_delete_hello_world.png similarity index 100% rename from Labs/backend_api/media/container_app_delete_hello_world.png rename to Labs/lab_5_backend_api/media/container_app_delete_hello_world.png diff --git a/Labs/backend_api/media/container_app_edit_and_deploy.png b/Labs/lab_5_backend_api/media/container_app_edit_and_deploy.png similarity index 100% rename from Labs/backend_api/media/container_app_edit_and_deploy.png rename to Labs/lab_5_backend_api/media/container_app_edit_and_deploy.png diff --git a/Labs/backend_api/media/container_app_failed_revision.png b/Labs/lab_5_backend_api/media/container_app_failed_revision.png similarity index 100% rename from Labs/backend_api/media/container_app_failed_revision.png rename to Labs/lab_5_backend_api/media/container_app_failed_revision.png diff --git a/Labs/backend_api/media/container_app_log_stream.png b/Labs/lab_5_backend_api/media/container_app_log_stream.png similarity index 100% rename from Labs/backend_api/media/container_app_log_stream.png rename to Labs/lab_5_backend_api/media/container_app_log_stream.png diff --git a/Labs/backend_api/media/container_app_overview.png b/Labs/lab_5_backend_api/media/container_app_overview.png similarity index 100% rename from Labs/backend_api/media/container_app_overview.png rename to Labs/lab_5_backend_api/media/container_app_overview.png diff --git a/Labs/backend_api/media/container_app_ready.png b/Labs/lab_5_backend_api/media/container_app_ready.png similarity index 100% rename from Labs/backend_api/media/container_app_ready.png rename to Labs/lab_5_backend_api/media/container_app_ready.png diff --git a/Labs/backend_api/media/container_deploy.png b/Labs/lab_5_backend_api/media/container_deploy.png similarity index 100% rename from Labs/backend_api/media/container_deploy.png rename to Labs/lab_5_backend_api/media/container_deploy.png diff --git a/Labs/backend_api/media/local_backend_docker_build.png b/Labs/lab_5_backend_api/media/local_backend_docker_build.png similarity index 100% rename from Labs/backend_api/media/local_backend_docker_build.png rename to Labs/lab_5_backend_api/media/local_backend_docker_build.png diff --git a/Labs/backend_api/media/local_backend_docker_push.png b/Labs/lab_5_backend_api/media/local_backend_docker_push.png similarity index 100% rename from Labs/backend_api/media/local_backend_docker_push.png rename to Labs/lab_5_backend_api/media/local_backend_docker_push.png diff --git a/Labs/backend_api/media/local_backend_docker_run.png b/Labs/lab_5_backend_api/media/local_backend_docker_run.png similarity index 100% rename from Labs/backend_api/media/local_backend_docker_run.png rename to Labs/lab_5_backend_api/media/local_backend_docker_run.png diff --git a/Labs/backend_api/media/local_backend_running_console.png b/Labs/lab_5_backend_api/media/local_backend_running_console.png similarity index 100% rename from Labs/backend_api/media/local_backend_running_console.png rename to Labs/lab_5_backend_api/media/local_backend_running_console.png diff --git a/Labs/backend_api/media/local_backend_swagger_ui.png b/Labs/lab_5_backend_api/media/local_backend_swagger_ui.png similarity index 100% rename from Labs/backend_api/media/local_backend_swagger_ui.png rename to Labs/lab_5_backend_api/media/local_backend_swagger_ui.png diff --git a/Labs/backend_api/media/local_backend_swagger_ui_ai_response.png b/Labs/lab_5_backend_api/media/local_backend_swagger_ui_ai_response.png similarity index 100% rename from Labs/backend_api/media/local_backend_swagger_ui_ai_response.png rename to Labs/lab_5_backend_api/media/local_backend_swagger_ui_ai_response.png diff --git a/Labs/backend_api/media/local_backend_swagger_ui_root_response.png b/Labs/lab_5_backend_api/media/local_backend_swagger_ui_root_response.png similarity index 100% rename from Labs/backend_api/media/local_backend_swagger_ui_root_response.png rename to Labs/lab_5_backend_api/media/local_backend_swagger_ui_root_response.png diff --git a/Labs/load_data/index.js b/Labs/load_data/index.js deleted file mode 100644 index 311e5db..0000000 --- a/Labs/load_data/index.js +++ /dev/null @@ -1,30 +0,0 @@ -require('dotenv').config(); -const { MongoClient } = require('mongodb'); - -async function main() { - const client = new MongoClient(process.env.MONGODB_URI); - try { - await client.connect(); - console.log('Connected to MongoDB'); - const db = client.db('cosmic_works'); - - - } catch (err) { - console.error(err); - } finally { - await client.close(); - console.log('Disconnected from MongoDB'); - } -} - -function cleanData(obj) { - cleaned = Object.fromEntries( - Object.entries(obj).filter(([key, _]) => !key.startsWith('_')) - ); - //rename id field to _id - cleaned["_id"] = cleaned["id"]; - delete cleaned["id"]; - return cleaned; -} - -main().catch(console.error); \ No newline at end of file diff --git a/README.md b/README.md index ec7c979..72f2f3e 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,34 @@ -# Azure Cosmos DB + Azure OpenAI Node.js Developer Guide - -1. [Introduction](/00_Introduction/README.md) -1. [Azure Overview](/01_Azure_Overview/README.md) -1. [Overview of Azure Cosmos DB](/02_Overview_Cosmos_DB/README.md) -1. [Overview of Azure OpenAI](/03_Overview_Azure_OpenAI/README.md) -1. [Overview of AI Concepts](/04_Overview_AI_Concepts/README.md) -1. [Explore the Azure OpenAI models and endpoints (console app)](/05_Explore_OpenAI_models/README.md) -1. [Provision Azure resources](/06_Provision_Azure_Resources/README.md) -1. [Create your first Cosmos DB project](/07_Create_First_Cosmos_DB_Project/README.md) -1. [Load data into Azure Cosmos DB API for MongoDB](/08_Load_Data/README.md) -1. [Use vector search on embeddings in vCore-based Azure Cosmos DB for MongoDB](/09_Vector_Search_Cosmos_DB/README.md) -1. [LangChain](/10_LangChain/README.md) -1. [Backend API](/11_Backend_API/README.md) -1. [Connect the chat user interface with the chatbot API](/12_User_Interface/README.md) -1. [Conclusion](/13_Conclusion/README.md) - -![Azure Cosmos DB + Azure OpenAI Node.js Developer Guide Architecture Diagram](/06_Provision_Azure_Resources/media/architecture.jpg) +# Azure DocumentDB + Azure OpenAI Node.js Developer Guide + +1. [Introduction](00_Introduction/README.md) +1. [Azure Overview](01_Azure_Overview/README.md) +1. [Overview of Azure Cosmos DB](02_Overview_Cosmos_DB/README.md) +1. [Overview of Azure OpenAI](03_Overview_Azure_OpenAI/README.md) +1. [Overview of AI Concepts](04_Overview_AI_Concepts/README.md) +1. [Explore the Azure OpenAI models and endpoints (console app)](05_Explore_OpenAI_models/README.md) +1. [Provision Azure resources](06_Provision_Azure_Resources/README.md) +1. [Create your first Cosmos DB project](07_Create_First_Cosmos_DB_Project/README.md) +1. [Load data into Azure Cosmos DB](08_Load_Data/README.md) +1. [Use vector search on embeddings in Azure Cosmos DB](09_Vector_Search_Cosmos_DB/README.md) +1. [LangChain](10_LangChain/README.md) +1. [Backend API](11_Backend_API/README.md) +1. [Connect the chat user interface with the chatbot API](12_User_Interface/README.md) +1. [Conclusion](13_Conclusion/README.md) + +![Azure DocumentDB + Azure OpenAI Node.js Developer Guide Architecture Diagram](06_Provision_Azure_Resources/media/architecture.jpg) ## Contributing -This project welcomes contributions and suggestions. Most contributions require you to agree to a +This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us -the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. +the rights to use your contribution. +For details, visit https://cla.opensource.microsoft.com. When you submit a pull request, a CLA bot will automatically determine whether you need to provide -a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions -provided by the bot. You will only need to do this once across all repos using our CLA. +a CLA and decorate the PR appropriately (e.g., status check, comment). +Simply follow the instructions +provided by the bot. +You will only need to do this once across all repos using our CLA. This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or @@ -33,8 +36,9 @@ contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additio ## Trademarks -This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft -trademarks or logos is subject to and must follow +This project may contain trademarks or logos for projects, products, or services. +Authorized use of Microsoft +trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.