The complexity around not enabling the video renderer before it
has a valid surface is because MediaCodecTrackRenderer supports
a "discard" mode where it pulls through and discards samples
without a decoder. This mode means that if the demo app were to
enable the renderer before supplying the surface, the renderer
could discard the first few frames prior to getting the surface,
meaning video rendering wouldn't happen until the following sync
frame.
To get a handle on complexity, I think we're better off just removing
support for this mode, which nicely decouples how the demo app
handles surfaces v.s. how it handles enabling/disabling renderers.
Reordering in the extractor isn't going to work well with the
optimizations I'm making there. This change moves sorting back
to the renderer, although keeps all of the renderer
simplifications. It's basically just moving where the sort
happens from one place to another.
I'm not really a fan of micro-optimizations, but given this method
scans through every H264 frame in the HLS case, it seems worthwhile.
The trick here is to examine the first 7 bits of the third byte
first. If they're not all 0s, then we know that we haven't found a
NAL unit, and also that we wont find one at the next two positions.
This allows the loop to increment 3 bytes at a time.
Speedup is around 60% on Art according to some ad-hoc benchmarking.
1. AdtsReader would previously copy all data through an intermediate
adtsBuffer. This change eliminates the additional copy step, and
instead copies directly into Sample objects.
2. PesReader would previously accumulate a whole packet by copying
multiple TS packets into an intermediate buffer. This change
eliminates this copy step. After the change, TS packet buffers
are propagated directly to PesPayloadReaders, which are required
to handle partial payload data correctly. The copy steps in the
extractor are simplified from:
DataSource->Ts_BitArray->Pes_BitArray->Sample->SampleHolder
To:
DataSource->Ts_BitArray->Sample->SampleHolder
Issue: #278
- TsExtractor is now based on ParsableByteArray rather than BitArray.
This makes is much clearer that, for the most part, data is byte
aligned. It will allow us to optimize TsExtractor without worrying
about arbitrary bit offsets.
- BitArray is renamed ParsableBitArray for consistency, and is now
exclusively for bit-stream level reading.
- There are some temporary methods in ParsableByteArray that should be
cleared up once the optimizations are in place.
Issue: #278
This is the start of a sequence of changes to fix the ref'd
github issue. Currently TsExtractor involves multiple memory
copy steps:
DataSource->Ts_BitArray->Pes_BitArray->Sample->SampleHolder
This is inefficient, but more importantly, the copy into
Sample is problematic, because Samples are of dynamically
varying size. The way we end up expanding Sample objects to
be large enough to hold the data being written means that we
end up gradually expanding all Sample objects in the pool
(which wastes memory), and that we generate a lot of GC churn,
particularly when switching to a higher quality which can
trigger all Sample objects to expand.
The fix will be to reduce the copy steps to:
DataSource->TsPacket->SampleHolder
We will track Pes and Sample data with lists of pointers into
TsPackets, rather than actually copying the data. We will
recycle these pointers.
The following steps are approximately how the refactor will
progress:
1. Start reducing use of BitArray. It's going to be way too
complicated to track bit-granularity offsets into multiple packets,
and allow reading across packet boundaries. In practice reads
from Ts packets are all byte aligned except for small sections,
so we'll move over to using ParsableByteArray instead, so we
only need to track byte offsets.
2. Move TsExtractor to use ParsableByteArray except for small
sections where we really need bit-granularity offsets.
3. Do the actual optimization.
Issue: #278
If the timesource track renderer ends, but other track renderers
haven't finished, the player would get stuck in a pending state.
This change enables automatic switching to the media clock in the
case that the timesource renderer has ended, which allows other
renderers to continue to play.
These may occur in VOD streams where a representation's data
is small enough not to require segmentation or an index. For
example subtitle files.
Issue: #268
SampleExtractor will initially only be implemented by FrameworkSampleExtractor
which delegates to a MediaExtractor, but eventually it will also be implemented
by additional extractors.
The sample extractor can be used as a source of samples via DefaultSampleSource.
Also added clamping to getSegmentNum in one case where
it was not already implemented, and defined this behavior
property in the getSegmentNum javadoc.
Issue: #262
Without this, the byte is cast as follows (in bits) if the top
byte is set:
10000010 -> 1000000000000000000000000000010
This works because we then always shift at least one bit left,
and only look at the bottom 8 bits of the result. It's confusing
though. It's clearer if the cast to int gives just adds zeros to
the front, like:
10000010 -> 0000000000000000000000010000010
- Workaround issue where video may freeze whilst audio continues
on some devices that have entered bad states.
- Fix wrap-around for playbacks lasting more than 27 hours.
- Target 4x the minimum specified by the framework.
- Impose a minimum duration (250ms).
- Impose a maximum duration (750ms, or the minimum
specified by the framework if that's larger).
I've removed the ability to specify the multiplication
factor, since the underlying implementation is getting more
complicated, and we should really be able to figure this out
internally.
* this fixes a bug when switching from HE-AAC 22050Hz to AAC 44100Hz (the AudioTrack was not reset and we were trying to send a bad number of bytes, triggering a "AudioTrack.write() called with invalid size" error)
* this also improves quality switches, making it almost seamless
Previously samples belonging to disabled tracks would just
accumulate in an arbitrarily long queue in TsExtractor. We
need to actively throw samples away from disabled tracks up
to the current playback position, so as to prevent this.
Issue: #174
I'm not sure exactly what the implications of this change are,
but I'd really hope that only one program in each stream is carrying
audio/video. For GoPro cameras, they expose the video stream in
the second program, for some reason.
Issue: #116
We've seen a few streams where this assertion fails. If you
just skip the packet, things appear to recover correctly in
all cases I've seen, so replacing failure with a warning.
- Handle read returning NOTHING_READ for AC-3 streams.
- Remove extra checks for the audio track being initialized.
- Call isInitialized() instead of checking audioTrack != null.
ac3Bitrate is set only after the first buffer is handled, which meant that
getting the playback position would cause a divide by zero before then.
When playing back AC-3 content, the ac3Bitrate will always be set after the
first buffer is handled, so return a 0 position if it is not set.
- Adds support for dash manifests that define SegmentTemplate
but no SegmentTimeline.
- Assumes that the device clock is correct when calculating which
segments to load. The final step here is to use the Utc timing
element in the DASH manifest to obtain an accurate client clock.
- Doesn't yet enforce that the client shouldn't load segments that
are in the future or behind the live window.
This fixes the referenced issue, except that the MPD parser
needs to actually parse out UUID and binary data for schemes
that we wish to support. Alternatively, it's easy to applications
to do this themselves by extending the parser and overriding
the parseContentProtection and buildContentProtection methods.
Github Issue: #119
It's cleaner to not inject data into the extractor only
so that it can be read out as though it were parsed from
the stream. This is also an incremental step towards
fixing Github issue #119.
Note: This adds support for the majority of DASH live streams,
however we do not yet correctly support live streams that rely
on UtcTimingElements in their manifests.
Issue: #52
Plus start to properly document the SmoothStreaming package.
Note that where the documentation is a little vague, this is
because the original SmoothStreaming documentation is equally
vague!
- Split sample pools for video/audio/misc, since the typical
required sample sizes are very different (and so it becomes
inefficient to use a sample sized for video to hold audio).
- Add TODO for further improvements.
Issue: #174
The timestamp scaling in SegmentBase.getSegmentTimeUs was
overflowing for some streams. Apply a similar trick to that
applied in the SmoothStreaming case to fix it.
- Move all three buffering constants to a single class (the
chunk source).
- Increase the target buffer to 40s for increased robustness
against temporary network blips.
- Make values configurable via the chunk source constructor.
- Treat captions as a text track for HLS. This allows them to
be enabled/disabled through the demo app UI.
Issue: #165
- Add options to switch abruptly at segment boundaries. Third
parties who guarantee keyframes at the start of segments will
want this, because it makes switching more efficient and hence
rebuffering less likely.
- Switch quality faster when performing a splicing switch (when
we detect that we need to switch variant, we now immediately
request the same segment as we did last time for the new variant,
rather than requesting one more segment for the old variant
before doing this.
1. Correctly replace the AES data source if IV changes.
2. Check the largest timestamp for being equal to MIN_VALUE, and
handle this case properly.
3. Clean up AES data source a little.
Issue: #162
The actual fix here is to not call discardExtractors in HlsSampleSource
whilst the loading thread that's pushing data into it is still running.
It's required to wait for that thread to have exited before doing this.
Issue: #159
Looking up a long in a HashSet<Long> auto boxes the long and leaves
it for the GC. As decodeOnly is relatively infrequent it's much
better to do a simple linear search in a List<Long>. That way
we can avoid boxing every incoming time stamp value. In the general
case this will be linear searching in an empty list, a very fast
operation.
Signed-off-by: Jonas Larsson <jonas@hallerud.se>
AudioTrack contains the portions of MediaCodecAudioTrackRenderer that handle the
platform AudioTrack instance, including synchronization (playback position
smoothing), non-blocking writes and releasing.
This refactoring should not affect the behavior of audio playback, and is in
preparation for adding an Ac3PassthroughAudioTrackRenderer that will use the
AudioTrack.
Passing uncropped dimensions to certain decoders will make them
output frames without proper cropping set.
Signed-off-by: Jonas Larsson <jonas@hallerud.se>
