Iahmad/migrate transcripts s3 #17605

irfanuddinahmad · 2018-03-02T12:16:15Z

Implementation of the design at:

https://openedx.atlassian.net/wiki/spaces/EDUCATOR/pages/162172272/Migration+from+contentstore+to+s3

edx-status-bot · 2018-03-02T13:13:27Z

Your PR has finished running tests.

edx-status-bot · 2018-03-07T11:23:21Z

Your PR has finished running tests.

edx-status-bot · 2018-03-07T14:00:13Z

Your PR has finished running tests.

mushtaqak

Did initial review. Things are looking good.

Miner comments on logging and single quotes across py file.

mushtaqak · 2018-03-07T13:32:54Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+        $ ./manage.py cms migrate_transcripts --all-courses --settings=devstack_docker
+        $ ./manage.py cms migrate_transcripts 'edX/DemoX/Demo_Course' --settings=devstack_docker
+    """
+    args = '<course_id course_id ...>'


Should we not have a --commit argument that will actually execute the command. If --commit is not provided the command will draft run and would print out the expected transcripts migrated?

mushtaqak · 2018-03-08T07:01:14Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+
+
+VIDEO_DICT_STAR = dict(
+    client_video_id="TWINKLE TWINKLE",


I would prefer single quotes here

mushtaqak · 2018-03-08T07:03:22Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+            data={'data': video_sample_xml}
+        )
+
+        save_to_store(SRT_FILEDATA, "subs_grmtran1.srt", 'text/srt', self.video_descriptor.location)


single quotes

mushtaqak · 2018-03-08T07:06:54Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+                **kwargs
+            )
+        except InvalidKeyError as exc:
+            raise CommandError(u'Invalid Course Key: ' + unicode(exc))


Invalid course key:

mushtaqak · 2018-03-08T07:11:54Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+                kwargs[key] = options[key]
+
+        try:
+            enqueue_async_migrate_transcripts_tasks(


Should we not check if the course exists with this key ? e.g
if not modulestore().get_course(course_key):

Added the check in def async_migrate_transcript

Added this check in def async_migrate_transcript

mushtaqak · 2018-03-08T07:44:27Z

cms/djangoapps/contentstore/tasks.py

+                                 revision=ModuleStoreEnum.RevisionOption.published_only, include_orphans=False):
+        all_videos.append(video)
+
+    for video in store.get_items(course_key, qualifiers={'category': 'video'},


This will be unneccessary if we do not provide revision argument

The all revision option does not seem to work ...

mushtaqak · 2018-03-08T07:44:44Z

cms/djangoapps/contentstore/tasks.py

+    return all_videos
+
+
+def is_transcript_content_srt(transcript_content):


docstring is missing

mushtaqak · 2018-03-08T07:45:01Z

cms/djangoapps/contentstore/tasks.py

+    try:
+        srt_subs_obj = SubRipFile.from_string(transcript_content.data.decode('utf-8-sig'))
+        if len(srt_subs_obj) > 0:
+            LOGGER.info("SRT file format detected")


Should we log this?

mushtaqak · 2018-03-08T09:27:11Z

cms/djangoapps/contentstore/tasks.py

+        transcript_name = args[2]
+        force_update = args[3]
+        result = None
+        LOGGER.info("Start migrating %s transcript", language_code)


lets use better text for this log as well

mushtaqak · 2018-03-08T09:27:39Z

cms/djangoapps/contentstore/tasks.py

+        try:
+            transcript_content = Transcript.asset(video.location, transcript_name, language_code)
+            if video.edx_video_id:
+                LOGGER.info("Found edx_video_id= %s via first fetch asset method", video.edx_video_id)


Add log prefix so as to make debugging easier

edx-status-bot · 2018-03-08T09:49:30Z

Your PR has finished running tests.

edx-status-bot · 2018-03-09T15:10:15Z

Your PR has finished running tests.

edx-status-bot · 2018-03-12T15:32:47Z

Your PR has finished running tests.

muhammad-ammar

@irfanuddinahmad I am done with first pass of the review. Let me know once feedback is addressed.

muhammad-ammar · 2018-03-13T07:22:59Z

cms/djangoapps/contentstore/tasks.py

+        video = args[0]
+        language_code = args[1]
+        transcript_name = args[2]
+        force_update = args[3]


we can do something like

video, language_code, transcript_name, force_update = args

muhammad-ammar · 2018-03-13T07:48:33Z

cms/djangoapps/contentstore/tasks.py

+                    ))
+            else:
+                LOGGER.info("video.sub is empty")
+            if any(other_lang_transcripts):


I think we don't need if other_lang_transcripts:. for loop will not run if other_lang_transcripts is empty.

I think code under if english_transcript: and for lang, name in other_lang_transcripts.items(): can be merged like below. As per my understanding we don't need separate handling for sub and transcripts in this case.

all_language_transcripts = dict({'en': video.sub}, **other_lang_transcripts) for lang, name in all_language_transcriptss.items(): ...

muhammad-ammar · 2018-03-13T09:34:12Z

cms/djangoapps/contentstore/tasks.py

+        try:
+            transcript_content, transcript_name, Transcript_mime_type = get_transcript_from_contentstore(
+                video, language_code, Transcript.SJSON)
+            if video.edx_video_id:


I think we can replace the whole if...else... with something like below

if not clean_video_id(video.edx_video_id): video.edx_video_id = create_external_video('external-video') video.save_with_metadata(user=User.objects.get(username='staff')) result = push_to_s3( video.edx_video_id, language_code, transcript_content, Transcript.SJSON, force_update )

muhammad-ammar · 2018-03-13T09:38:14Z

cms/djangoapps/contentstore/tasks.py

+
+
+@task(base=PersistOnFailureTask)
+def async_migrate_transcript(*args, **kwargs):


what about def async_migrate_transcript(course_key, **kwargs):?

muhammad-ammar · 2018-03-13T09:38:47Z

cms/djangoapps/contentstore/tasks.py

+    kwargs = {
+        'force_update': force_update,
+        'commit': commit
+              }


indentation is bit off

muhammad-ammar · 2018-03-13T10:00:45Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        self.assertEqual(len(languages), 0)
+
+        # now call migrate_transcripts command and check the transcript availability
+        call_command('migrate_transcripts', unicode(self.course.id))


I think we should verify the output of command in this case to check if the correct message is returned.

The command output is checked by the following test
def test_migrate_transcripts_logging

Yes we should check to have been migrated transcripts here.

muhammad-ammar · 2018-03-13T10:01:14Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        save_to_store(SRT_FILEDATA, 'subs_grmtran1.srt', 'text/srt', self.video_descriptor.location)
+        save_to_store(CRO_SRT_FILEDATA, 'subs_croatian1.srt', 'text/srt', self.video_descriptor.location)
+
+    def test_migrated_transcripts_count(self):


please add docstring to explain what we are testing/verifying in this test.

muhammad-ammar · 2018-03-13T10:01:30Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        languages = api.get_available_transcript_languages(self.video_descriptor.edx_video_id)
+        self.assertEqual(len(languages), 2)
+
+    def test_migrated_transcripts_without_commit(self):


same comment as above.

The command output is checked by the following test
def test_migrate_transcripts_logging

muhammad-ammar · 2018-03-13T10:03:29Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        """
+        Test migrating transcripts
+        """
+        translations = self.video_descriptor.available_translations(self.video_descriptor.get_transcripts_info())


Don't we need to pass include_val_transcripts to get_transcripts_info and available_translations?

@Qubad786 what do you say?

muhammad-ammar · 2018-03-13T10:05:21Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        call_command('migrate_transcripts', unicode(self.course.id), '--force-update', '--commit')
+
+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'hr'))
+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'ge'))


can we do the above in a loop?

the command arguments change for subsequent calls ... a loop will decrease readability

muhammad-ammar · 2018-03-13T10:08:07Z

@Qubad786 You also need to look into this PR.

edx-status-bot · 2018-03-13T15:29:10Z

Your PR has finished running tests.

edx-status-bot · 2018-03-14T11:25:17Z

Your PR has finished running tests.

edx-status-bot · 2018-03-14T15:26:34Z

Your PR has finished running tests.

edx-status-bot · 2018-03-16T15:17:28Z

Your PR has finished running tests.

edx-status-bot · 2018-03-19T10:36:38Z

Your PR has finished running tests.

irfanuddinahmad · 2018-03-21T11:42:42Z

cms/djangoapps/contentstore/tasks.py

+                    ))
+            LOGGER.info("[Transcript migration] process for video %s ended", video.location)
+        callback = task_status_callback.s()
+        status = chord(sub_tasks)(callback)


@nasthagiri I have used a celery task for each video inside a celery task for each course. Do you think this could create too many celery tasks, which could potentially swamp the task queue? The benefit of a 'task per video' could be a more granular failure tracking and retry.

@irfanuddinahmad That should be fine given our other recent usage of celery tasks:

we create a celery task for each learner that we send an email to via ACE

we create a celery task for each problem submission by a learner

However, you should let devops know so they are aware in case they see spikes in tasks at certain times. Also, the team will want to monitor this the first time you roll it out.

Things to consider:

Is there a time_limit placed on the task? If your task is calling the network, you'll want to timeout appropriately (and exponentially backoff) in case of a persistent network failure.

Are you using our celery-utils, which provides functionality for logging and persistent retries? You may find this useful if you want to persist failures (after celery's retries give up) and automatically retry them via a Jenkins job.

You can see some of this in action in the grades tasks code.

@nasthagiri Thanks. Very useful feedback (as usual! :) )
I am already using the PersistOnFailureTask ... let me try the
celeryutils.chordable_django_backend and LoggedPersistOnFailureTask also

mushtaqak

@irfanuddinahmad Done with second pass of the review. This is looking good.

mushtaqak · 2018-03-22T06:35:02Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+                 u'Without this flag, the command will return the transcripts discovered for migration ',
+        )
+
+    def handle(self, *args, **options):


Method docstring?

mushtaqak · 2018-03-22T06:39:33Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+            raise CommandError('At least one course or --all-courses must be specified.')
+
+        kwargs = {}
+        for key in ('all_courses', 'force_update', 'commit'):


May be we can write this as:

kwargs = {key: options[key] for key in ['all_courses', 'force_update', 'commit'] if options.get(key)}

mushtaqak · 2018-03-22T06:41:40Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+
+1
+00:00:02,720 --> 00:00:05,430
+Ja, ich spreche Deutsch


Lets also add one more line so as to check unicode characters.

2 00:00:6,500 --> 00:00:08,600 可以用“我不太懂艺术但我知道我喜欢什么”做比喻.

mushtaqak · 2018-03-22T06:41:56Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+    client_video_id='TWINKLE TWINKLE',
+    duration=42.0,
+    edx_video_id='test_edx_video_id',
+    status="upload",


nit: single quotes

mushtaqak · 2018-03-22T06:43:30Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+
+    def test_invalid_course(self):
+        with self.assertRaises(CommandError):
+            call_command('migrate_transcripts', "invalid-course")


nit: 'invalid-course'

mushtaqak · 2018-03-22T07:56:31Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        self.assertEqual(len(languages), 0)
+
+        # now call migrate_transcripts command and check the transcript availability
+        call_command('migrate_transcripts', unicode(self.course.id))


Yes we should check to have been migrated transcripts here.

mushtaqak · 2018-03-22T07:56:59Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        languages = api.get_available_transcript_languages(self.video_descriptor.edx_video_id)
+        self.assertEqual(len(languages), 2)
+
+    def test_migrated_transcripts_without_commit(self):


Missing docstring

mushtaqak · 2018-03-22T07:58:50Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        # now call migrate_transcripts command again and check the transcript availability
+        call_command('migrate_transcripts', unicode(self.course.id), '--commit')
+
+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'hr'))


we should also check the count before and after the command run.

mushtaqak · 2018-03-22T08:01:16Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'hr'))
+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'ge'))
+
+    def test_migrate_transcripts_logging(self):


We haven't checked for any exception logs. We need to check them as well in this test or a separate test

I think this was missed.

mushtaqak · 2018-03-22T08:05:56Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+            call_command('migrate_transcripts', "invalid-course")
+
+
+class MigrateTranscripts(ModuleStoreTestCase):


MigrateTranscriptsTest

edx-status-bot · 2018-03-22T10:17:02Z

Your PR has finished running tests.

edx-status-bot · 2018-03-22T16:14:06Z

Your PR has finished running tests.

edx-status-bot · 2018-03-26T08:17:33Z

Your PR has finished running tests.

edx-status-bot · 2018-03-26T17:14:13Z

Your PR has finished running tests.

edx-status-bot · 2018-03-27T08:20:40Z

Your PR has finished running tests.

edx-status-bot · 2018-03-27T13:18:15Z

Your PR has finished running tests.

edx-status-bot · 2018-03-27T15:44:25Z

Your PR has finished running tests.

edx-status-bot · 2018-03-30T14:58:23Z

Your PR has finished running tests.

irfanuddinahmad · 2018-04-02T12:26:08Z

@nasthagiri Kindly advise regarding the trigger for our video transcript management command. Some options are discussed in the comments of the following issue:
https://openedx.atlassian.net/browse/ENT-641
Devops seem to have a certain preference for Django admin UI. However, Jenkins and DSL could be another option.

edx-status-bot · 2018-04-02T16:57:47Z

Your PR has finished running tests.

edx-status-bot · 2018-04-04T11:20:14Z

Your PR has finished running tests.

nasthagiri · 2018-04-04T11:48:16Z

@irfanuddinahmad https://openedx.atlassian.net/browse/ENT-641 takes me to a SuccessFactors ticket. Did you intend another ticket number?

irfanuddinahmad · 2018-04-04T11:59:25Z

"@irfanuddinahmad https://openedx.atlassian.net/browse/ENT-641 takes me to a SuccessFactors ticket. Did you intend another ticket number?"
@nasthagiri Kindly have a look at Fred's comments. I mentioned this ticket since they also had a management command that was to be run on Production and Stage.

nasthagiri · 2018-04-04T16:43:22Z

@irfanuddinahmad ah. Thanks.

I agree with Fred that there are valid reasons to not have Jenkins jobs be fully parameterized. To Backfill Grades, we created a management command for the following reasons:

We wanted the Open edX community to also have the ability to backfill their own grades.
We needed this support (to recompute all grades in a course) even in the long-run (post-migration).

We then created a Jenkins Job that would be manually triggered to run the management command.

Following devOps practices, we configured the management command via Django Admin.

Now, for video migrations, you can do the same thing we did for Grades:
Have a Jenkins job, but configured via Django Admin.

Or, you can go with Option #1 specified in the ticket:
Create a new Django Admin action-button, also configured via Django Admin.

irfanuddinahmad · 2018-04-05T10:29:56Z

@nasthagiri Thanks. We will have a Jenkins job configured via Django Admin.

edx-status-bot · 2018-04-05T12:16:30Z

Your PR has finished running tests.

nasthagiri · 2018-04-05T12:54:24Z

@irfanuddinahmad Ok. Please inform your devops contact person so s/he is aware of your path forward.

edx-status-bot · 2018-04-05T15:49:19Z

Your PR has finished running tests.

edx-status-bot · 2018-04-06T08:16:36Z

Your PR has finished running tests.

edx-status-bot · 2018-04-06T10:43:14Z

Your PR has finished running tests.

mushtaqak · 2018-04-09T12:42:41Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

            default=DEFAULT_ALL_COURSES,
            help=u'Migrates transcripts to S3 for all courses.'
        )
+        parser.add_argument(


What are these settings?

mushtaqak · 2018-04-09T12:43:27Z

openedx/core/djangoapps/video_config/models.py

+    """
+    force_update = BooleanField(default=False)
+    commit = BooleanField(default=False)
+    course_ids = TextField(


Don't we need all_courses option here?

mushtaqak · 2018-04-09T12:47:08Z

openedx/core/djangoapps/video_config/models.py

+
+    force_update = BooleanField(
+        default=False,
+        help_text="Flag to update transcripts in django storage even if already present."


Flag to force migrate transcripts for the requested courses, overwrite if already present.

mushtaqak · 2018-04-09T12:48:15Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+            '--course-id', '--course_id',
+            dest='course_ids',
+            action='append',
+            help=u'Migrates transcripts for list of courses.'


nit: Migrates transcripts for the list of courses

mushtaqak · 2018-04-09T12:50:24Z

cms/djangoapps/contentstore/management/commands/migrate_transcripts.py

+            action='store_true',
+            default=DEFAULT_COMMIT,
+            help=u'Commits the discovered video transcripts to django storage. '
+                 u'Without this flag, the command will return the transcripts discovered for migration '


nit: Remove extra space in transcripts discovered for migration probably add a dot 😃

mushtaqak · 2018-04-09T12:54:44Z

cms/djangoapps/contentstore/management/commands/tests/test_migrate_transcripts.py

+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'hr'))
+        self.assertTrue(api.is_transcript_available(self.video_descriptor.edx_video_id, 'ge'))
+
+    def test_migrate_transcripts_logging(self):


I think this was missed.

mushtaqak · 2018-04-09T12:55:58Z

cms/djangoapps/contentstore/tasks.py

+def async_migrate_transcript_subtask(self, *args, **kwargs):
+    #pylint: disable=unused-argument
+    """
+         Migrates a transcript of a given video in a course as a new celery task.


indentation is a bit off here

mushtaqak · 2018-04-09T13:00:46Z

cms/djangoapps/contentstore/tasks.py

+                dict({'file_format': file_format}),
+                ContentFile(transcript_content)
+            )
+            LOGGER.info("[Transcript migration] Push_to_storage %s for %s with create_or_update method",


nit: LOGGER.info("[Transcript migration] save_transcript_to_storage: transcript video %s is %s over-written", '' if result else 'not', edx_video_id)

mushtaqak · 2018-04-09T13:01:22Z

cms/djangoapps/contentstore/tasks.py

+                file_format,
+                ContentFile(transcript_content)
+            )
+            LOGGER.info("[Transcript migration] Push_to_storage %s for %s with create method", result, edx_video_id)


might need to change to save_transcript_to_storage

mushtaqak · 2018-04-09T13:01:27Z

cms/djangoapps/contentstore/tasks.py

+            LOGGER.info("[Transcript migration] Push_to_storage %s for %s with create method", result, edx_video_id)
+        return result
+    except ValCannotCreateError as err:
+        LOGGER.exception("[Transcript migration] Push_to_storage_failed: %s", err)


might need to change to save_transcript_to_storage

edx-status-bot · 2018-04-11T16:23:05Z

Your PR has finished running tests.

muhammad-ammar · 2018-04-12T06:51:43Z

cms/djangoapps/contentstore/tasks.py

above two variables are unused?

these are from the branched off master code ...

muhammad-ammar · 2018-04-12T07:00:49Z

cms/djangoapps/contentstore/tasks.py

I think we can do clean_video_id only once like this.

edx_video_id = clean_video_id(video.edx_video_id) if not edx_video_id: video.edx_video_id = create_external_video('external-video') video.save_with_metadata(user=User.objects.get(username='staff')) if edx_video_id: result = save_transcript_to_storage( edx_video_id, language_code, transcript_content, Transcript.SJSON, force_update )

And we need to remove clean_video_id from save_transcript_to_storage as we not making any decision on edx_video_id in it.

mushtaqak

Looks good to me, just one outstanding question remaining though about all_courses option in model.

After that's addressed. Let's rebase and merge. Good work 👍

mushtaqak · 2018-04-12T07:56:44Z

openedx/core/djangoapps/video_config/models.py

Should we not have all_courses option here?

edx-status-bot · 2018-04-12T09:11:05Z

Your PR has finished running tests.

muhammad-ammar

👍

edx-status-bot · 2018-04-12T11:37:32Z

Your PR has finished running tests.

edx-status-bot · 2018-04-12T12:40:39Z

Your PR has finished running tests.

edx-pipeline-bot · 2018-04-12T15:39:05Z

EdX Release Notice: This PR has been deployed to the staging environment in preparation for a release to production on Friday, April 13, 2018.

edx-pipeline-bot · 2018-04-13T16:55:32Z

EdX Release Notice: This PR has been deployed to the production environment.

edx-pipeline-bot · 2018-04-13T20:31:13Z

EdX Release Notice: This PR has been rolled back from the production environment.

irfanuddinahmad self-assigned this Mar 2, 2018

irfanuddinahmad requested review from Qubad786, muhammad-ammar and mushtaqak March 2, 2018 12:16

mushtaqak suggested changes Mar 8, 2018

View reviewed changes

irfanuddinahmad force-pushed the iahmad/migrate_transcripts_S3 branch from 201bf55 to 049ef73 Compare March 9, 2018 14:12

muhammad-ammar suggested changes Mar 13, 2018

View reviewed changes

irfanuddinahmad force-pushed the iahmad/migrate_transcripts_S3 branch from a59ff2a to e6d5216 Compare March 19, 2018 09:32

irfanuddinahmad commented Mar 21, 2018

View reviewed changes

mushtaqak suggested changes Mar 22, 2018

View reviewed changes

irfanuddinahmad force-pushed the iahmad/migrate_transcripts_S3 branch from 8a5cf1f to 239b4ad Compare March 27, 2018 14:32

mushtaqak suggested changes Apr 11, 2018

View reviewed changes

muhammad-ammar reviewed Apr 12, 2018

View reviewed changes

mushtaqak approved these changes Apr 12, 2018

View reviewed changes

openedx/core/djangoapps/video_config/models.py Outdated

Copy link

Contributor

mushtaqak Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not have all_courses option here?

muhammad-ammar approved these changes Apr 12, 2018

View reviewed changes

irfanuddinahmad force-pushed the iahmad/migrate_transcripts_S3 branch from 7bad68d to fabdb1d Compare April 12, 2018 11:05

New command to migrate transcripts to S3

3e426ac

irfanuddinahmad force-pushed the iahmad/migrate_transcripts_S3 branch from fabdb1d to 3e426ac Compare April 12, 2018 11:59

irfanuddinahmad merged commit e424e04 into master Apr 12, 2018

irfanuddinahmad deleted the iahmad/migrate_transcripts_S3 branch April 12, 2018 12:46

		return all_videos


		def is_transcript_content_srt(transcript_content):



		@task(base=PersistOnFailureTask)
		def async_migrate_transcript(args, *kwargs):

		call_command('migrate_transcripts', "invalid-course")


		class MigrateTranscripts(ModuleStoreTestCase):

Iahmad/migrate transcripts s3 #17605

Iahmad/migrate transcripts s3 #17605

Uh oh!

Conversation

irfanuddinahmad commented Mar 2, 2018

Uh oh!

edx-status-bot commented Mar 2, 2018

Uh oh!

edx-status-bot commented Mar 7, 2018

Uh oh!

edx-status-bot commented Mar 7, 2018

Uh oh!

mushtaqak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edx-status-bot commented Mar 8, 2018

Uh oh!

edx-status-bot commented Mar 9, 2018

Uh oh!

edx-status-bot commented Mar 12, 2018

Uh oh!

muhammad-ammar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irfanuddinahmad Mar 21, 2018 •

edited

Loading

nasthagiri Mar 21, 2018 •

edited

Loading