TrainTrack

A Telegram bot for controlling NN training process remotely!
Report Bug · Request Feature · Contact Me

Table of Contents

• About the Project
  • Features
  • Built with
• Getting Started
  • Prerequisites
  • Installation
  • Implementation
• Usage
• Roadmap
• Contributing
• License
• Contact
• Acknowledgements

About the Project

TrainTrack is a Telegram bot that helps you track, visualize and control your neural network training process on various platforms using python-telegram-bot wrapper for Telegram Bot API. TrainTrack is developed for PyTorch framework but can be used with TensorFlow and Keras as well.

Origin of the bot and it's implementation on TenserFlow, Keras can be found on @eyalzk's GitHub repo.

An example implementation of the bot on PyTorch MNIST example is in the source files under the filename pytorch_mnist_example.py. Read the source code thoroughly for better understanding its implementation on your neural network trainer. The example is well commented and points out each line of code you should implement in order for TrainTrack to function properly.

Features

• Get updates on loss, accuracy with personalized messages

Show message commands

• Control the learning rate

Show learning rate commands

• Query the latest metrics

Show status command

• Get loss, accuracy convergence plots

Show plot command

• Stop the training process

Show stop command

• Limit access to a specific Telegram user ID

Built with

• Python

Getting Started

Follow these guidelines to get TrainTrack up and running!

Prerequisites

• python-telegram-bot | Required for Telegram bot to function
• matplotlib | Required for sending plots to the user
• torch, torchvision | Required if you want to run pytorch_mnist_example.py

pip install python-telegram-bot
pip install matplotlib
pip install torch torchvision

Installation

1. Make sure you have installed required prerequisites

2. Clone this repository

git clone https://github.com/batuhanfaik/TrainTrack.git

3. Copy traintrack.py to your trainer's source file destination

cd TrainTrack/
cp traintrack.py /<your>/<training_dir>/<source_files>/

4. Create a free Telegram bot using the Telegram app. Just follow these instructions provided in the official Telegram bot documentation.

5. Take a note of the given token. It will be used later in the Implementation section.

   • Optional: You can follow this article to learn your Telegram user ID. Your user ID can be used to limit the access of TrainTrack only to yourself

Implementation

• Import bot in your trainer source file

Show code

Following piece of code is all you need to import TrainTrack into your project

# Import TrainTrack Bot
from traintrack import TrainTrack

telegram_token = "TOKEN"  # TrainTrack's token
# User id is optional and can be kept as None.
# However highly recommended as it limits the access to you alone.
telegram_user_id = None  # Telegram user id (integer):
# Create a TrainTrack Bot instance
TrainTrack = TrainTrack(token=telegram_token, user_id=telegram_user_id)
# Activate the bot
TrainTrack.activate_bot()

• Add TrainTrack's update_epoch() method, /stop and learning rate controlling commands in the training loop

Show code

Following piece of code is needed to be placed in your training loop

# Update the epoch variable in TrainTrack in order to keep track of
# the current epoch
TrainTrack.update_epoch(epoch)
# Force break epoch loop when the user stops training
if TrainTrack.stop_train_flag:
    break
# Manually control learning rate using TrainTrack
if TrainTrack.learning_rate is not None:
    for param_group in optimizer.param_groups:
        param_group["lr"] = TrainTrack.learning_rate

• Using TrainTrack's update_prereport(), update_message(), add_status(), clr_status() methods, you can control the messages that will be sent out by TrainTrack

• Using TrainTrack's cumulate_<train/test>_<loss/acc>() methods, you can cumulate your losses and accuracies for the plot

• Add TrainTrack's stop_bot() method and exit condition handlers after the training loop

Show code

Following piece of code is needed to be placed after your training loop

# Exit conditions handling for TrainTrack
# Notifies the user whether the training has terminated or finished after completing all epochs
if TrainTrack.stop_train_flag:
    print("Training stopped by {}!".format(TrainTrack.name))
    TrainTrack.send_message("Training stopped by {}!".format(TrainTrack.name))
else:
    print("Training complete. {} out!".format(TrainTrack.name))
    TrainTrack.send_message("Training complete. {} out!".format(TrainTrack.name))
# Stop TrainTrack Bot instance at the end of training
TrainTrack.stop_bot()

Usage

Each command is shown in the Features section. Please refer to it since there is no documentation available for this project yet.

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/new_feature)
3. Commit your Changes (git commit -m 'Added this new_feature')
4. Push to the Branch (git push origin feature/new_feature)
5. Open a Pull Request

Keep in mind that pull requests are always welcome!

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Batuhan Faik Derinbay

LinkedIn | Email | This Project

Acknowledgements

• telegrad
• python-telegram-bot