Large Classes Experience Assignment Timeout

[pancakereport]

Heroku has a strict 30 second timeout: https://devcenter.heroku.com/articles/request-timeout. So even though gunicorn is set to have a 60 second timeout, heroku's timeout still applies (and I see 503 / H12 logging and the request is closed before it's completed). 

From logging, I'd guess that assignment for 61a midterm 2 takes an extra 10-15 seconds beyond the 30 second timeout limit because I can see it complete and send back a 302. But because the 30 second timeout was already hit, the request is closed and the redirect can't happen. 

The current workaround is assigning half the class seats and then assigning the other half.

Would appreciate input @cycomachead on if you think adjusting the algorithm is the best next step, or if we should look into using asynchronous processing (like celery) where the assignment POST request queues the job and then immediately returns 302/202 to the user who gets taken to a "Processing" page that updates when the job is complete.

[cycomachead]

Yes, I think we want some background job framework.

Alternatively, we could explore GCP/AWS hosting too, but IMO almost everything roster-related should be a background job. (syncing students, assigning students, emailing all students, etc.)

It's not clear to me why assigning 1000 people to ~3000 seats actually takes so long. I do feel like this should be able to complete w/in 30 secs but I haven't verified that code. I suspect there is a bunch of database stuff that could be batched as well.