fixed race condition in recording, Recording file is now moved, inste… by pchlupacek · Pull Request #200 · RestComm/media-core

pchlupacek · 2016-06-25T12:27:30Z

as discussed in #128

…ad copy in place

hrosa · 2016-06-27T14:17:14Z

During my tests I got the following exception:

15:13:01,132 ERROR [AudioRecorderImpl] Error writing to file
java.nio.file.NoSuchFileException: /Users/hrosa/Telestax/mobicents/restcomm/binaries/Restcomm-JBoss-AS7-7.6.0.873/standalone/deployments/restcomm.war/recordings/RE8abfb69ded5a4000b4550ae557a784c6.wav~ -> /Users/hrosa/Telestax/mobicents/restcomm/binaries/Restcomm-JBoss-AS7-7.6.0.873/standalone/deployments/restcomm.war/recordings/RE8abfb69ded5a4000b4550ae557a784c6.wav
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:451)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1347)
at org.mobicents.media.server.impl.resource.audio.AudioRecorderImpl.commitRecording(AudioRecorderImpl.java:413)
at org.mobicents.media.server.impl.resource.audio.AudioRecorderImpl.deactivate(AudioRecorderImpl.java:181)
at org.mobicents.media.server.mgcp.pkg.au.PlayRecord.terminateRecordPhase(PlayRecord.java:308)
at org.mobicents.media.server.mgcp.pkg.au.PlayRecord.terminate(PlayRecord.java:380)
at org.mobicents.media.server.mgcp.pkg.au.PlayRecord.cancel(PlayRecord.java:217)
at org.mobicents.media.server.mgcp.controller.Request.cancel(Request.java:132)
at org.mobicents.media.server.mgcp.tx.cmd.NotificationRequestCmd$Request.perform(NotificationRequestCmd.java:121)
at org.mobicents.media.server.scheduler.Task.run(Task.java:122)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

hrosa · 2016-06-30T10:10:52Z

Another issue appeared during load testing:

19:39:54,280 ERROR [AudioRecorderImpl] Error writing to file
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:99)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:273)
at org.mobicents.media.server.impl.resource.audio.AudioRecorderImpl.commitRecording(AudioRecorderImpl.java:405)
at org.mobicents.media.server.impl.resource.audio.AudioRecorderImpl.deactivate(AudioRecorderImpl.java:181)
at org.mobicents.media.server.mgcp.pkg.au.PlayRecord.terminateRecordPhase(PlayRecord.java:308)
at org.mobicents.media.server.mgcp.pkg.au.PlayRecord.terminate(PlayRecord.java:380)
at org.mobicents.media.server.mgcp.pkg.au.PlayRecord.cancel(PlayRecord.java:217)
at org.mobicents.media.server.mgcp.controller.Request.cancel(Request.java:132)
at org.mobicents.media.server.mgcp.tx.cmd.NotificationRequestCmd$Request.perform(NotificationRequestCmd.java:121)
at org.mobicents.media.server.scheduler.Task.run(Task.java:122)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Both issues seem to be extremely rare, still they point to concurrency problems in the recorder.

pchlupacek · 2016-06-30T10:13:39Z

@hrosa right. do you have any ideas when the different methods on recorder are called? I mean would be really good to know if the fix shall be proper.

hrosa · 2016-06-30T10:24:49Z

I think this is because you still have cleanup logic in the checkIn() method.
I will advise again to try commenting that method until we get the lifecycle of these components done right.

without much analysis, I can tell you

recorder is taken from pool (checkout())
recorder is activated (activate())
then multiple invocations of onMediaTransfer() during recording operation
recording terminates and deactivate() is invoked
recorder is put back into the pool (checkIn)

I was naive enough to believe it was safe to implement checkIn method and do cleanup there, believing that since object is being put back into the pool then it was not being used anymore.
It seems that is not the case: maybe the last few bytes are still being written in a different thread or some other concurrent operation is still ongoing.

Another problem i noticed during my load tests (wrt recording use case) is that the resulting files tend to decrease in size sometimes. Recording quality is good though.
Maybe its a symptom of the problem I described: a new call picks up a pooled recorder that was still finishing writing operation and stops.

hrosa · 2016-06-30T10:33:07Z

Couple of notes:

we must somehow guarantee that recorder is not placed back into the pool until the recording operation is absolutely completed.
Notice that a recording operation may terminate prematurely because of many different factors (DTMF key pressed, no voice detected in x seconds, user hangs up, etc).

We need to solidify concurrent operations in recorder taking into account these factors.

hrosa · 2016-06-30T10:45:48Z

@pchlupacek I think we need to consider this behaviour:

Heartbeat task, which runs in different thread, is responsible for the timeouts I mentioned before (no voice, max recording duration, etc) and submits a KillRecording task to the scheduler, which will be executed in yet another thread:

https://github.com/RestComm/mediaserver/blob/master/resources/recorder/src/main/java/org/mobicents/media/server/impl/resource/audio/AudioRecorderImpl.java#L567
https://github.com/RestComm/mediaserver/blob/master/resources/recorder/src/main/java/org/mobicents/media/server/impl/resource/audio/AudioRecorderImpl.java#L573
https://github.com/RestComm/mediaserver/blob/master/resources/recorder/src/main/java/org/mobicents/media/server/impl/resource/audio/AudioRecorderImpl.java#L581

Then the kill recording executes deactivate() and completes the File writing operation:
https://github.com/RestComm/mediaserver/blob/master/resources/recorder/src/main/java/org/mobicents/media/server/impl/resource/audio/AudioRecorderImpl.java#L521

Keep also in mind that the Media Group may also deactivate() the recorder upon release:
https://github.com/RestComm/mediaserver/blob/master/controls/mgcp/src/main/java/org/mobicents/media/server/mgcp/endpoint/MediaGroup.java#L210
The media group will then release it back to the pool, where checkIn() is invoked.

I think this is where concurrency issue between deactivate() and checkIn() happens.

Any thoughts on how to improve this?

pchlupacek · 2016-06-30T12:18:59Z

@hrosa will review today, and see how this can be done.

pchlupacek · 2016-07-01T04:16:00Z

@hrosa My first thought would be to just toss away any Recorder once used. That way we can prevent anything to happen after Recorder is deactivated. If that isn't the solution, then I think only issue we have there is that we have to guarantee to null any references to file before the file move operation actually starts.

As you suggest, we will have the cleanup code only on one place (I remove that checkin check). Furthermore I am thinking to introduce an AtomicRef to guard about perhaps multiple concurrent invocations of the deactivate code.

Any thoughts? Suggestions?

pchlupacek · 2016-07-06T09:02:25Z

@hrosa just pushed the update. This is sort of re-work, and I believe shall be correct and safe for now. (At least cannot think of scenarios where concurrent invocations of deactivate or checkin will mess.

I have introduced the Sink, that is tossed away with each deactivate call and created when the file is set. Also this guarantees, that there cannot be write operations while the sink is making its commit, in case there were some leftover references writing to the recorder.

I run 10k call through it and no issue.

lmk what do you think

…ng-fix-2 # Conflicts: # resources/recorder/src/main/java/org/mobicents/media/server/impl/resource/audio/AudioRecorderImpl.java

hrosa · 2016-07-12T16:04:41Z

nicely done @pchlupacek thank you!

fixed race condition in recording, Recording file is now moved, inste…

bad4e56

…ad copy in place

deruelle added the Peer Review label Jun 25, 2016

pchlupacek added 2 commits June 25, 2016 14:28

simplified initial logic of file creation

0abc92b

Added missing attributes to create target file

105688a

hrosa self-assigned this Jun 27, 2016

hrosa added this to the 4.2.0 milestone Jun 27, 2016

hrosa modified the milestones: 4.3.0, 4.2.0 Jun 27, 2016

hrosa mentioned this pull request Jun 30, 2016

Implement proper FSM for Audio Recorder #212

Open

Introduced RecordingFileSink with safe concurrent behaviour

2171e16

pchlupacek added 4 commits July 6, 2016 11:10

Merge branch 'master' of github.com:RestComm/mediaserver into recordi…

616eb28

…ng-fix-2 # Conflicts: # resources/recorder/src/main/java/org/mobicents/media/server/impl/resource/audio/AudioRecorderImpl.java

Fixed the potential race in JitterBuffer

6de9ae3

Lock in read

87bd49c

Added licence header

c90976e

hrosa merged commit 6cb0a49 into RestComm:master Jul 12, 2016

hrosa removed the Peer Review label Jul 12, 2016

hrosa mentioned this pull request Sep 29, 2016

Recorder thread blocks when writing to file #223

Open

hrosa modified the milestones: 5.1.0, 6.0.0 Feb 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixed race condition in recording, Recording file is now moved, inste…#200

fixed race condition in recording, Recording file is now moved, inste…#200
hrosa merged 8 commits into
RestComm:masterfrom
Spinoco:recording-fix-2

pchlupacek commented Jun 25, 2016

Uh oh!

hrosa commented Jun 27, 2016

Uh oh!

hrosa commented Jun 30, 2016

Uh oh!

pchlupacek commented Jun 30, 2016

Uh oh!

hrosa commented Jun 30, 2016 •

edited

Loading

Uh oh!

hrosa commented Jun 30, 2016

Uh oh!

hrosa commented Jun 30, 2016 •

edited

Loading

Uh oh!

pchlupacek commented Jun 30, 2016

Uh oh!

pchlupacek commented Jul 1, 2016

Uh oh!

pchlupacek commented Jul 6, 2016 •

edited

Loading

Uh oh!

hrosa commented Jul 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pchlupacek commented Jun 25, 2016

Uh oh!

hrosa commented Jun 27, 2016

Uh oh!

hrosa commented Jun 30, 2016

Uh oh!

pchlupacek commented Jun 30, 2016

Uh oh!

hrosa commented Jun 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hrosa commented Jun 30, 2016

Uh oh!

hrosa commented Jun 30, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pchlupacek commented Jun 30, 2016

Uh oh!

pchlupacek commented Jul 1, 2016

Uh oh!

pchlupacek commented Jul 6, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hrosa commented Jul 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hrosa commented Jun 30, 2016 •

edited

Loading

hrosa commented Jun 30, 2016 •

edited

Loading

pchlupacek commented Jul 6, 2016 •

edited

Loading