From 89241ac3f1dd8ceab1247ffffa0f332e721c17cb Mon Sep 17 00:00:00 2001 From: joffinjoy Date: Fri, 15 Mar 2024 15:45:40 +0530 Subject: [PATCH 1/4] Added 2.5 Docs --- src/api-doc/notification-2.5-setup.md | 173 ++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 src/api-doc/notification-2.5-setup.md diff --git a/src/api-doc/notification-2.5-setup.md b/src/api-doc/notification-2.5-setup.md new file mode 100644 index 00000000..87365b96 --- /dev/null +++ b/src/api-doc/notification-2.5-setup.md @@ -0,0 +1,173 @@ +# ShikshaLokam Notification Service Documentation + +## System Requirements + +- **Operating System:** Ubuntu 22 +- **Node.js:** v20 +- **PostgreSQL:** 16 +- **Citus:** 12.1 + +## Installations + +### Install Node.js LTS + +Refer to the [NodeSource distributions installation scripts](https://github.com/nodesource/distributions#installation-scripts) for Node.js installation. + +```bash +$ curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash - &&\ +sudo apt-get install -y nodejs +``` + +### Install Build Essential + +```bash +$ sudo apt-get install build-essential +``` + +### Install PM2 + +Refer to [How To Set Up a Node.js Application for Production on Ubuntu 22.04](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-22-04). + +**Run the following command** + +```bash +$ sudo npm install pm2@latest -g +``` + +## Setting up Repository + +### Clone the notification repository to /opt/backend directory + +```bash +opt/backend$ git clone -b develop-2.5 --single-branch "https://github.com/ELEVATE-Project/notification.git" +``` + +### Install Npm packages from src directory + +```bash +backend/notification/src$ sudo npm i +``` + +### Create .env file in src directory + +```bash +notification/src$ sudo nano .env +``` + +Copy-paste the following env variables to the `.env` file: + +```env +# Notification Service Config + +# Port on which service runs +APPLICATION_PORT=3002 + +# Application environment +APPLICATION_ENV=development + +# Route after the base URL +APPLICATION_BASE_URL=/notification/ + +# Kafka endpoint +KAFKA_HOST="localhost:9092" + +# Kafka topic name +KAFKA_TOPIC="dev.notification" + +# Kafka consumer group id +KAFKA_GROUP_ID="elevate-notification" + +# Sendgrid API key +SENDGRID_API_KEY="SG.asd9f87a9s8d7f." + +# Sendgrid sender email address +SENDGRID_FROM_MAIL="no-reply@some.org" + +# Api doc URL +API_DOC_URL= "/notification/api-doc" + +INTERNAL_ACCESS_TOKEN="internal_access_token" +ERROR_LOG_LEVEL='silly' +DISABLE_LOG=false +DEV_DATABASE_URL=postgres://shikshalokam:slpassword@localhost:9700/elevate_notification + +ZEST_ENV= "ZEST_ENV" +created_time= "2023-12-29T17:04:19.017783534Z" +custom_metadata= null +destroyed=false +version=8 +``` + +Save and exit. + +## Setting up Databases + +**Log into the postgres user** + +```bash +sudo su postgres +``` + +**Log into psql** + +```bash +psql -p 9700 +``` + +**Create a database user/role:** + +```sql +CREATE USER shikshalokam WITH ENCRYPTED PASSWORD 'slpassword'; +``` + +**Create the elevate_notification database** + +```sql +CREATE DATABASE elevate_notification; +GRANT ALL PRIVILEGES ON DATABASE elevate_notification TO shikshalokam; +\c elevate_notification +GRANT ALL ON SCHEMA public TO shikshalokam; +``` + +## Running Migrations To Create Tables + +**Exit the postgres user account and install sequelize-cli globally** + +```bash +$ sudo npm i sequelize-cli -g +``` + +**Navigate to the src folder of notification service and run sequelize-cli migration command:** + +```bash +notification/src$ npx sequelize-cli db:migrate +``` + +**Now all the tables must be available in the Citus databases** + +## Start the Services + +Navigate to the src folder of notification service and run pm2 start command: + +```bash +notification/src$ pm2 start app.js -i 2 --name elevate-notification +``` + +#### Run pm2 ls command + +```bash +$ pm2 ls +``` + +Output should look like this (Sample output, might slightly differ in your installation): + +```bash +┌────┬─────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐ +│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │ +├────┼─────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤ +│ 19 │ elevate-notification │ default │ 1.0.0 │ cluster │ 88026 │ 47h │ 0 │ online │ 0% │ 113.2mb │ jenkins │ disabled │ +│ 20 │ elevate-notification │ default │ 1.0.0 │ cluster │ 88036 │ 47h │ 0 │ online │ 0% │ 80.3mb │ jenkins │ disabled │ +└────┴─────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘ +``` + +This concludes the service and dependency setup. From 17a3afd3370275fe4200e3580400e907b8b2dc8c Mon Sep 17 00:00:00 2001 From: joffinjoy Date: Fri, 15 Mar 2024 15:57:02 +0530 Subject: [PATCH 2/4] Minor Typo Fix --- src/api-doc/notification-2.5-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/api-doc/notification-2.5-setup.md b/src/api-doc/notification-2.5-setup.md index 87365b96..b713acaf 100644 --- a/src/api-doc/notification-2.5-setup.md +++ b/src/api-doc/notification-2.5-setup.md @@ -145,7 +145,7 @@ notification/src$ npx sequelize-cli db:migrate **Now all the tables must be available in the Citus databases** -## Start the Services +## Start the Service Navigate to the src folder of notification service and run pm2 start command: From 0dc8b27258e6ed5ac8ac13b9c5fa3c85fb45e668 Mon Sep 17 00:00:00 2001 From: joffinjoy Date: Fri, 15 Mar 2024 17:10:22 +0530 Subject: [PATCH 3/4] Added Kafka Setup To Doc --- src/api-doc/notification-2.5-setup.md | 112 ++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/src/api-doc/notification-2.5-setup.md b/src/api-doc/notification-2.5-setup.md index b713acaf..f753bb84 100644 --- a/src/api-doc/notification-2.5-setup.md +++ b/src/api-doc/notification-2.5-setup.md @@ -24,6 +24,118 @@ sudo apt-get install -y nodejs $ sudo apt-get install build-essential ``` +### Install Kafka + +Refer to [Kafka Ubuntu 22.04 setup guide](https://www.fosstechnix.com/install-apache-kafka-on-ubuntu-22-04-lts/) + +1. Install OpenJDK 11: + + ```bash + $ sudo apt install openjdk-11-jdk + ``` + +2. Download and extract Kafka: + + ```bash + $ sudo wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgz + $ sudo tar xzf kafka_2.12-3.5.0.tgz + $ sudo mv kafka_2.12-3.5.0 /opt/kafka + ``` + +3. Configure Zookeeper: + + ```bash + $ sudo nano /etc/systemd/system/zookeeper.service + ``` + + Paste the following lines into the `zookeeper.service` file: + + ```ini + /etc/systemd/system/zookeeper.service + [Unit] + Description=Apache Zookeeper service + Documentation=http://zookeeper.apache.org + Requires=network.target remote-fs.target + After=network.target remote-fs.target + + [Service] + Type=simple + ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties + ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh + Restart=on-abnormal + + [Install] + WantedBy=multi-user.target + ``` + + Save and exit. + +4. Reload systemd: + + ```bash + $ sudo systemctl daemon-reload + ``` + +5. Configure Kafka: + + ```bash + $ sudo nano /etc/systemd/system/kafka.service + ``` + + Paste the following lines into the `kafka.service` file: + + ```ini + [Unit] + Description=Apache Kafka Service + Documentation=http://kafka.apache.org/documentation.html + Requires=zookeeper.service + + [Service] + Type=simple + Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" + ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties + ExecStop=/opt/kafka/bin/kafka-server-stop.sh + + [Install] + WantedBy=multi-user.target + ``` + + Save and exit. + +6. Reload systemd: + + ```bash + $ sudo systemctl daemon-reload + ``` + +7. Start Zookeeper: + + ```bash + $ sudo systemctl start zookeeper + ``` + + Check status: + + ```bash + $ sudo systemctl status zookeeper + ``` + + Zookeeper service status should be shown as active (running). + +8. Start Kafka: + + ```bash + $ sudo systemctl start kafka + ``` + + Check status: + + ```bash + $ sudo systemctl status kafka + ``` + + Kafka status should be shown as active (running). + ### Install PM2 Refer to [How To Set Up a Node.js Application for Production on Ubuntu 22.04](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-22-04). From 0a11ccd407af20bf20b0bc8c60bada9720e286a3 Mon Sep 17 00:00:00 2001 From: joffinjoy Date: Fri, 15 Mar 2024 17:54:03 +0530 Subject: [PATCH 4/4] Added Setup Guide To README --- README.md | 291 ++++++++++++++++++++++---- src/api-doc/notification-2.5-setup.md | 1 + 2 files changed, 256 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index e3661576..c7529084 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,14 @@ The Mentoring building block enables effective mentoring interactions between me
+# System Requirements + +- **Operating System:** Ubuntu 22 +- **Node.js:** v20 +- **PostgreSQL:** 16 +- **Citus:** 12.1 +- **Apache Kafka:** 3.5.0 + # Setup Options Elevate notification services can be setup in local using two methods: @@ -108,75 +116,286 @@ Elevate notification services can be setup in local using two methods: **Expectation**: Run single service with existing local dependencies in host (**Non-Docker Implementation**). -### Steps +## Installations + +### Install Node.js LTS + +Refer to the [NodeSource distributions installation scripts](https://github.com/nodesource/distributions#installation-scripts) for Node.js installation. + +```bash +$ curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash - &&\ +sudo apt-get install -y nodejs +``` + +### Install Build Essential -1. Install required tools & dependencies +```bash +$ sudo apt-get install build-essential +``` - Install any IDE (eg: VScode) +### Install Kafka - Install Nodejs: https://nodejs.org/en/download/ +Refer to [Kafka Ubuntu 22.04 setup guide](https://www.fosstechnix.com/install-apache-kafka-on-ubuntu-22-04-lts/) -2. Clone the **Notification service** repository. +1. Install OpenJDK 11: + ```bash + $ sudo apt install openjdk-11-jdk ``` - git clone https://github.com/ELEVATE-Project/notification.git + +2. Download and extract Kafka: + + ```bash + $ sudo wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgz + $ sudo tar xzf kafka_2.12-3.5.0.tgz + $ sudo mv kafka_2.12-3.5.0 /opt/kafka ``` -3. Add **.env** file to the project directory +3. Configure Zookeeper: + + ```bash + $ sudo nano /etc/systemd/system/zookeeper.service + ``` - Create a **.env** file in **src** directory of the project and copy these environment variables into it. + Paste the following lines into the `zookeeper.service` file: + ```ini + /etc/systemd/system/zookeeper.service + [Unit] + Description=Apache Zookeeper service + Documentation=http://zookeeper.apache.org + Requires=network.target remote-fs.target + After=network.target remote-fs.target + + [Service] + Type=simple + ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties + ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh + Restart=on-abnormal + + [Install] + WantedBy=multi-user.target ``` - # Notification Service Config - #Port on which service runs - APPLICATION_PORT = 3000 + Save and exit. - #Application environment - APPLICATION_ENV = development +4. Reload systemd: - #Route after base url - APPLICATION_BASE_URL = /notification/ + ```bash + $ sudo systemctl daemon-reload + ``` - #Kafka endpoint - KAFKA_HOST = "localhost:9092" +5. Configure Kafka: - #kafka topic name - KAFKA_TOPIC ="testTopic" + ```bash + $ sudo nano /etc/systemd/system/kafka.service + ``` - #kafka consumer group id - KAFKA_GROUP_ID = "notification" + Paste the following lines into the `kafka.service` file: - #sendgrid api key - SENDGRID_API_KEY = "SG.sdssd.dsdsd.XVSDGFEBGEB.sddsd" + ```ini + [Unit] + Description=Apache Kafka Service + Documentation=http://kafka.apache.org/documentation.html + Requires=zookeeper.service - #sendgrid sender email address - SENDGRID_FROM_MAIL = "test@gmail.com" + [Service] + Type=simple + Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" + ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties + ExecStop=/opt/kafka/bin/kafka-server-stop.sh + [Install] + WantedBy=multi-user.target ``` -4. Install Npm packages + Save and exit. + +6. Reload systemd: + ```bash + $ sudo systemctl daemon-reload ``` - ELEVATE/notification/src$ npm install + +7. Start Zookeeper: + + ```bash + $ sudo systemctl start zookeeper ``` -5. Start Notification server + Check status: + ```bash + $ sudo systemctl status zookeeper ``` - ELEVATE/notification/src$ npm start + + Zookeeper service status should be shown as active (running). + +8. Start Kafka: + + ```bash + $ sudo systemctl start kafka ``` - + Check status: -
+ ```bash + $ sudo systemctl status kafka + ``` + + Kafka status should be shown as active (running). + +### Install PM2 + +Refer to [How To Set Up a Node.js Application for Production on Ubuntu 22.04](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-22-04). + +**Run the following command** + +```bash +$ sudo npm install pm2@latest -g +``` + +## Setting up Repository + +### Clone the notification repository to /opt/backend directory + +```bash +opt/backend$ git clone -b develop-2.5 --single-branch "https://github.com/ELEVATE-Project/notification.git" +``` + +### Install Npm packages from src directory + +```bash +backend/notification/src$ sudo npm i +``` + +### Create .env file in src directory + +```bash +notification/src$ sudo nano .env +``` + +Copy-paste the following env variables to the `.env` file: + +```env +# Notification Service Config + +# Port on which service runs +APPLICATION_PORT=3002 + +# Application environment +APPLICATION_ENV=development -# Tech stack +# Route after the base URL +APPLICATION_BASE_URL=/notification/ -- Node - 16.0.0 -- Kafka - 3.1.0 -- Jest - 28.1.1 -- MongoDB - 4.1.4 +# Kafka endpoint +KAFKA_HOST="localhost:9092" + +# Kafka topic name +KAFKA_TOPIC="dev.notification" + +# Kafka consumer group id +KAFKA_GROUP_ID="elevate-notification" + +# Sendgrid API key +SENDGRID_API_KEY="SG.asd9f87a9s8d7f." + +# Sendgrid sender email address +SENDGRID_FROM_MAIL="no-reply@some.org" + +# Api doc URL +API_DOC_URL= "/notification/api-doc" + +INTERNAL_ACCESS_TOKEN="internal_access_token" +ERROR_LOG_LEVEL='silly' +DISABLE_LOG=false +DEV_DATABASE_URL=postgres://shikshalokam:slpassword@localhost:9700/elevate_notification + +ZEST_ENV= "ZEST_ENV" +created_time= "2023-12-29T17:04:19.017783534Z" +custom_metadata= null +destroyed=false +version=8 +``` + +Save and exit. + +## Setting up Databases + +**Log into the postgres user** + +```bash +sudo su postgres +``` + +**Log into psql** + +```bash +psql -p 9700 +``` + +**Create a database user/role:** + +```sql +CREATE USER shikshalokam WITH ENCRYPTED PASSWORD 'slpassword'; +``` + +**Create the elevate_notification database** + +```sql +CREATE DATABASE elevate_notification; +GRANT ALL PRIVILEGES ON DATABASE elevate_notification TO shikshalokam; +\c elevate_notification +GRANT ALL ON SCHEMA public TO shikshalokam; +``` + +## Running Migrations To Create Tables + +**Exit the postgres user account and install sequelize-cli globally** + +```bash +$ sudo npm i sequelize-cli -g +``` + +**Navigate to the src folder of notification service and run sequelize-cli migration command:** + +```bash +notification/src$ npx sequelize-cli db:migrate +``` + +**Now all the tables must be available in the Citus databases** + +## Start the Service + +Navigate to the src folder of notification service and run pm2 start command: + +```bash +notification/src$ pm2 start app.js -i 2 --name elevate-notification +``` + +#### Run pm2 ls command + +```bash +$ pm2 ls +``` + +Output should look like this (Sample output, might slightly differ in your installation): + +```bash +┌────┬─────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐ +│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │ +├────┼─────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤ +│ 19 │ elevate-notification │ default │ 1.0.0 │ cluster │ 88026 │ 47h │ 0 │ online │ 0% │ 113.2mb │ jenkins │ disabled │ +│ 20 │ elevate-notification │ default │ 1.0.0 │ cluster │ 88036 │ 47h │ 0 │ online │ 0% │ 80.3mb │ jenkins │ disabled │ +└────┴─────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘ +``` + +This concludes the service and dependency setup. + + + +
# Run tests diff --git a/src/api-doc/notification-2.5-setup.md b/src/api-doc/notification-2.5-setup.md index f753bb84..8783eada 100644 --- a/src/api-doc/notification-2.5-setup.md +++ b/src/api-doc/notification-2.5-setup.md @@ -6,6 +6,7 @@ - **Node.js:** v20 - **PostgreSQL:** 16 - **Citus:** 12.1 +- **Apache Kafka:** 3.5.0 ## Installations