MediaElement is buffering when MediaStreamSource has enough samples (Windows 8.1 store app)

MediaElement is buffering when MediaStreamSource has enough samples
(Windows 8.1 store app)

I have a Windows 8.1 app (NOT Silverlight or WPF) which plays custom
audio/video stream (aac/h264 file in ASF format). I parse the stream and
supply samples to MediaStreamSource for MediaElement (in MSS
SampleRequested event). The stream plays fine for a few seconds and then
pauses (MediaElement state changes from playing to buffering) then it
plays again and stops again etc. I verified that MediaStreamSource has
enough samples to supply to MediaElement (I never have to use deferral).
The network conditions are ideal (I stream the media file via ISS on the
same PC where I run the app). If I play only the audio or only the video
portion of the file, it works fine with no pauses. Increasing MSS
Buffertime reduces the problem. I'd appreciate any advice on how to
troubleshoot this problem. Thanks.