The complexity around not enabling the video renderer before it
has a valid surface is because MediaCodecTrackRenderer supports
a "discard" mode where it pulls through and discards samples
without a decoder. This mode means that if the demo app were to
enable the renderer before supplying the surface, the renderer
could discard the first few frames prior to getting the surface,
meaning video rendering wouldn't happen until the following sync
frame.
To get a handle on complexity, I think we're better off just removing
support for this mode, which nicely decouples how the demo app
handles surfaces v.s. how it handles enabling/disabling renderers.
Reordering in the extractor isn't going to work well with the
optimizations I'm making there. This change moves sorting back
to the renderer, although keeps all of the renderer
simplifications. It's basically just moving where the sort
happens from one place to another.
I'm not really a fan of micro-optimizations, but given this method
scans through every H264 frame in the HLS case, it seems worthwhile.
The trick here is to examine the first 7 bits of the third byte
first. If they're not all 0s, then we know that we haven't found a
NAL unit, and also that we wont find one at the next two positions.
This allows the loop to increment 3 bytes at a time.
Speedup is around 60% on Art according to some ad-hoc benchmarking.
1. AdtsReader would previously copy all data through an intermediate
adtsBuffer. This change eliminates the additional copy step, and
instead copies directly into Sample objects.
2. PesReader would previously accumulate a whole packet by copying
multiple TS packets into an intermediate buffer. This change
eliminates this copy step. After the change, TS packet buffers
are propagated directly to PesPayloadReaders, which are required
to handle partial payload data correctly. The copy steps in the
extractor are simplified from:
DataSource->Ts_BitArray->Pes_BitArray->Sample->SampleHolder
To:
DataSource->Ts_BitArray->Sample->SampleHolder
Issue: #278
- TsExtractor is now based on ParsableByteArray rather than BitArray.
This makes is much clearer that, for the most part, data is byte
aligned. It will allow us to optimize TsExtractor without worrying
about arbitrary bit offsets.
- BitArray is renamed ParsableBitArray for consistency, and is now
exclusively for bit-stream level reading.
- There are some temporary methods in ParsableByteArray that should be
cleared up once the optimizations are in place.
Issue: #278
This is the start of a sequence of changes to fix the ref'd
github issue. Currently TsExtractor involves multiple memory
copy steps:
DataSource->Ts_BitArray->Pes_BitArray->Sample->SampleHolder
This is inefficient, but more importantly, the copy into
Sample is problematic, because Samples are of dynamically
varying size. The way we end up expanding Sample objects to
be large enough to hold the data being written means that we
end up gradually expanding all Sample objects in the pool
(which wastes memory), and that we generate a lot of GC churn,
particularly when switching to a higher quality which can
trigger all Sample objects to expand.
The fix will be to reduce the copy steps to:
DataSource->TsPacket->SampleHolder
We will track Pes and Sample data with lists of pointers into
TsPackets, rather than actually copying the data. We will
recycle these pointers.
The following steps are approximately how the refactor will
progress:
1. Start reducing use of BitArray. It's going to be way too
complicated to track bit-granularity offsets into multiple packets,
and allow reading across packet boundaries. In practice reads
from Ts packets are all byte aligned except for small sections,
so we'll move over to using ParsableByteArray instead, so we
only need to track byte offsets.
2. Move TsExtractor to use ParsableByteArray except for small
sections where we really need bit-granularity offsets.
3. Do the actual optimization.
Issue: #278
If the timesource track renderer ends, but other track renderers
haven't finished, the player would get stuck in a pending state.
This change enables automatic switching to the media clock in the
case that the timesource renderer has ended, which allows other
renderers to continue to play.
These may occur in VOD streams where a representation's data
is small enough not to require segmentation or an index. For
example subtitle files.
Issue: #268
SampleExtractor will initially only be implemented by FrameworkSampleExtractor
which delegates to a MediaExtractor, but eventually it will also be implemented
by additional extractors.
The sample extractor can be used as a source of samples via DefaultSampleSource.
Also added clamping to getSegmentNum in one case where
it was not already implemented, and defined this behavior
property in the getSegmentNum javadoc.
Issue: #262
Without this, the byte is cast as follows (in bits) if the top
byte is set:
10000010 -> 1000000000000000000000000000010
This works because we then always shift at least one bit left,
and only look at the bottom 8 bits of the result. It's confusing
though. It's clearer if the cast to int gives just adds zeros to
the front, like:
10000010 -> 0000000000000000000000010000010
- Workaround issue where video may freeze whilst audio continues
on some devices that have entered bad states.
- Fix wrap-around for playbacks lasting more than 27 hours.
- Target 4x the minimum specified by the framework.
- Impose a minimum duration (250ms).
- Impose a maximum duration (750ms, or the minimum
specified by the framework if that's larger).
I've removed the ability to specify the multiplication
factor, since the underlying implementation is getting more
complicated, and we should really be able to figure this out
internally.
Previously samples belonging to disabled tracks would just
accumulate in an arbitrarily long queue in TsExtractor. We
need to actively throw samples away from disabled tracks up
to the current playback position, so as to prevent this.
Issue: #174
I'm not sure exactly what the implications of this change are,
but I'd really hope that only one program in each stream is carrying
audio/video. For GoPro cameras, they expose the video stream in
the second program, for some reason.
Issue: #116
We've seen a few streams where this assertion fails. If you
just skip the packet, things appear to recover correctly in
all cases I've seen, so replacing failure with a warning.
- Handle read returning NOTHING_READ for AC-3 streams.
- Remove extra checks for the audio track being initialized.
- Call isInitialized() instead of checking audioTrack != null.
ac3Bitrate is set only after the first buffer is handled, which meant that
getting the playback position would cause a divide by zero before then.
When playing back AC-3 content, the ac3Bitrate will always be set after the
first buffer is handled, so return a 0 position if it is not set.
- Adds support for dash manifests that define SegmentTemplate
but no SegmentTimeline.
- Assumes that the device clock is correct when calculating which
segments to load. The final step here is to use the Utc timing
element in the DASH manifest to obtain an accurate client clock.
- Doesn't yet enforce that the client shouldn't load segments that
are in the future or behind the live window.
This fixes the referenced issue, except that the MPD parser
needs to actually parse out UUID and binary data for schemes
that we wish to support. Alternatively, it's easy to applications
to do this themselves by extending the parser and overriding
the parseContentProtection and buildContentProtection methods.
Github Issue: #119
It's cleaner to not inject data into the extractor only
so that it can be read out as though it were parsed from
the stream. This is also an incremental step towards
fixing Github issue #119.
Note: This adds support for the majority of DASH live streams,
however we do not yet correctly support live streams that rely
on UtcTimingElements in their manifests.
Issue: #52
Plus start to properly document the SmoothStreaming package.
Note that where the documentation is a little vague, this is
because the original SmoothStreaming documentation is equally
vague!
- Split sample pools for video/audio/misc, since the typical
required sample sizes are very different (and so it becomes
inefficient to use a sample sized for video to hold audio).
- Add TODO for further improvements.
Issue: #174
