Adding Data

[sjaybush18]

• Database integration with the webscraping scripts for consistent automation
• May require updating and optimizing the webscraping scripts
• Ensuring up-to-date professor, department, and opportunity data
• Ensure these changes and updates work with the existing backend and frontend

[swadekar]

• Implement error handling in webscraping scripts.
• Document webscraping process and dependencies.
• Schedule periodic updates for data consistency.
• Test integration with the database for compatibility.
• Document schema changes for new data fields.
• Optimize webscraping performance.

[RafaelCenzano]

As a note, this should be done as a job where we scrape, process data, and input it into the database (ETL)

[RafaelCenzano]

Webscraping is not required, it can be useful for auto updating but can also break easily. Preference is creating ETL scripts/methods and separately setting up either webscraping or just downloading of data manually for example courses should change regularly enough to waste time now automating that process, departments similarly aren't updated very often

Course data: #67
Department data: #164