Fix short term reference picture list parsing. Before this change, `deltaPocS0`
was derived by adding one to the value of the syntax element
`delta_poc_s0_minus1`, but (maybe surprising) the specification actually says
that `DeltaPocS0[stRpsIdx][i]` should be assigned the negation
`-(delta_poc_s0_minus1[i] + 1)` on the first iteration, then that value added
to the previous value on previous iterations. See equations (7-67) to (7-70) in
the 2021-08 version of the H.265/HEVC specification.
Also read the number of long term reference pictures once rather than on every
loop iteration (subsection 7.3.2.2.1).
PiperOrigin-RevId: 551852999
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).
This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)
- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
- We bypass the input manager
- The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
- The generated frame is sent to texture output and in turn saved to bitmaps
- We then compare the generated bitmap with the goldens
PiperOrigin-RevId: 551795770
Move shared test logic to the test runner.
This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.
PiperOrigin-RevId: 551536438
1. For codecs that change timestamps, we need to use greater
or equal instead of just equal for comparison.
2. We should only make this comparison once a value has been
set.
3. The largest queue timestamp isn't updated before using it
for this detection, meaning that we also mark the
second-to-last sample as the last one.
PiperOrigin-RevId: 550901978
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.
PiperOrigin-RevId: 550894935
We currently pass in the time at which the stream originally started,
but newly enabled renderers should get the current playback position
instead.
PiperOrigin-RevId: 550894630
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.
Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.
Regression tests reproduce failure without fix, but pass with it.
PiperOrigin-RevId: 550853714
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.
PiperOrigin-RevId: 550839156
Instead of `text/x-exoplayer-cues`, we will use `application/x-media3-cues`.
The prefix should be `application` not `text` since the encoded form is arbitrary bytes, not necessarily text. The name should not reference `exoplayer`, since the Media3 Extractors (which are not part of `exoplayer`) produce this format.
PiperOrigin-RevId: 550181852
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.
onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors
PiperOrigin-RevId: 549969933
The VideoFrameProcessingTaskExecutor already handles wrapping
GlUtil.GlException errors into VideoFrameProcessingExceptions.
Therefore, stop wrapping this ourselves, so that errors are
attributed to the correct line number with less indirection.
PiperOrigin-RevId: 549959258
Alters the OverlayShaderProgram implementation to support rotations, however we need to apply the transformations separately in order for them to work as expected so the matrix is removed from the interface in favour of explicit methods.
Adds a rotation test to ensure this ability doesn't regress
PiperOrigin-RevId: 549890847
The `MediaNotificationManager` registers an internal controller
to each session. This change marks this controller through its
connection hints and provides an API for apps to hide
implementation details of the marking.
Issue: androidx/media#389
PiperOrigin-RevId: 549712768
`TtmlDecoder` which used to be `SimpleSubtitleDecoder` will now be called `TtmlParser` and implement SubtitleParser interface. For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `TtmlParser` instance.
PiperOrigin-RevId: 549700490
It is not part of the `SubtitleParser` interface's promise to give events for all changes to do with `Cue` objects. So while in the past, we would have `endTimeUs` of one Cue event being the `startTimeUs` of the next one, now we have that encoded in `durationUs` and can skip event with empty `Cues`.
PiperOrigin-RevId: 549629157
This tracker aims to replicate the behavior of a specific codec to
ensure MediaCodecRenderer correctly detects stream and output
format transitions. The class was needed because MediaCodecRenderer
made assumptions about codec behavior this codec did not fulfil (in
particular, changing timestamps and number of samples).
Since then, MediaCodecRenderer was made more robust to this kind of
codec behavior in general and currently has no assumptions that
require any special handling of this codec. This means we can remove
the workaround completely.
PiperOrigin-RevId: 549610989
This is no longer flaky because WebVTT subtitles are decoded on the
loading thread (enabled in this test with
`defaultExractorsFactory.setTextTrackTranscodingEnabled(true)` - and
supported for WebVTT since f0f24aa0d4).
PiperOrigin-RevId: 549594291
The class tries to be flexible to support as many different input
and codec behavior combinations as possible. But so far it didn't
spell out its remaining assumptions and explicit non-assumptions,
making it hard to know which behavior to rely on.
PiperOrigin-RevId: 549589347
Updated `ExoTrackSelection` to provide the most recent bitrate estimate, enabling the inclusion of measured throughput (mtp) as a CMCD-Request field in Common Media Client Data (CMCD) logging.
Additionally, made changes to the `checkArgument` methods in `CmcdLog` to prevent the use of default values in certain cases.
#minor-release
PiperOrigin-RevId: 549369529
`WebvttDecoder` which used to be `SimpleSubtitleDecoder` will now be called `WebvttParser` and implement `SubtitleParser` interface.
For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `WebvttParser` instance. `WebvttSubtitle` will still be used behind the scenes to handle overlapping `Cues`.
PiperOrigin-RevId: 549298733
The end-to-end test output for the overlapping SRT and SSA subtitles
is currently incorrect. They will be fixed in a future change that
updates `TextRenderer` to support this overlap.
The 'extra' samples visible in the extractor test output files are
'empty cue list' samples produced by `SsaParser`. They will go away
when this implementation is updated to remove this behaviour and rely
on `CuesWithTiming.durationUs` instead (the 'empty list' behaviour is
not required by the `SubtitleParser` interface).
PiperOrigin-RevId: 549264593
The main demo app was still targeting API 29 to avoid scoped storage
restrictions. It is now updated to 34 (like the rest of the demo apps)
and handles scoped storage as it should handle it.
More specifically:
- We need to request READ_MEDIA_... permissions instead of
READ_EXTERNAL_STORAGE from API33.
- The legacy scoped storage opt-out can be removed
- READ_MEDIA_... permissions don't allow arbitrary file access
if the file doesn't end in a typical media file extension, so
this change adds a remark on the guide page to place samples
in the app-specific directory.
- We also don't have to request permissions for the app-specific
directories.
- Custom json files can't be placed in arbitray local locations
because they don't end in a media file extension, as there is
no way we can request a permission to load them. This means we
can remove the storage request logic in SampleChooserActivity.
Issue: google/ExoPlayer#6045
PiperOrigin-RevId: 549252474
*** Original commit ***
Rollback of b69b33206e
*** Original commit ***
Mark output sample as decode-only based on start time
We currently do the same check on the input timestamps and
expect the output timestamps to match. Some codecs produce
samples with modified timestamps and the logic is a lot safer
when the comparison with the start time is done on the output
side of the codec.
Issue: google/ExoPlayer#11000
***
***
PiperOrigin-RevId: 549019403
This ensures the DownloadService stays functional on Android 14
where defining this type is required. On Android 14 and above,
the app also needs to define the DATA_SYNC permission, which is
added to the demo app as well. In the future, this service type
will no longer be supported and DownloadService needs to be
rewritten with another background scheduling framework.
Issue: google/ExoPlayer#11239
PiperOrigin-RevId: 548994842
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.
Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.
PiperOrigin-RevId: 548994584
This was done because it was deemed correct to only start at timestamp
zero when the code was originally written. However, in case of
prerolling from a keyframe, many samples will get the same timestamp,
which is not correct and interferes with downstream logic
that deals with timestamps.
PiperOrigin-RevId: 548986160
This fixes two things in one go:
1. In order to indicate 'end of a cue' **without** relying on
`CuesWithTiming.durationUs`, `SubripParser` should have been emitting
an empty `List<Cue>` instead of the current behaviour of a
single-item list with `cue.text=""`.
2. There's no need for the empty cue (or cue list), we can use
`durationUs` to indicate the end of each list of cues.
There's no real need to ever have a `Cue` with `text=""`, so also
deprecate `Cue.EMPTY`.
PiperOrigin-RevId: 548938874
Reading a discontinuity from a media period indicates that a position
reset is required. As part of this event, the media period may need
further loading (e.g in a MergingMediaPeriod where one stream reported
a discontinuity and the other need to reload from this position).
This currently fails if the media periods was already fully loaded and
we started loading further items in the playlist. As a result, playback
is stuck forever. We can fix this by detecting that further loading is
needed and resetting the loading period to the current one.
The existing MergingPlaylistPlaybackTest already covers this case
reliably, because it combines all the right preconditions (merging
source, clipping to get a discontinuity and a playlist).
PiperOrigin-RevId: 548735177
This is a step towards adding general support for overlapping
subtitles in these formats (and others), both muxed and sideloaded:
* Issue: google/ExoPlayer#10295
* Issue: google/ExoPlayer#4794
This change adds these files to the end-to-end playback tests too, but
the subtitle track is currently disabled because renderer-side subtitle
parsing causes flaky tests (due to an uncontrolled thread in
`SimpleSubtitleDecoder`). The subtitle track will be re-enabled in
a follow-up change when loading-side subtitle parsing is added (so the
tests will no longer be flaky). At this point the overlapping subtitles
**still** won't be supported end-to-end, but a second change will
resolve this will changes in `TextRenderer` - which will change the
end-to-end playback dumps to reflect the overlapping subtitles.
PiperOrigin-RevId: 548705032
For now, just identical timestamps. A follow-up CL will allow for different
framerates. Prior tests already send identical timestamp frames, so no new
tests were added
PiperOrigin-RevId: 548676759
`EGL14` was added in API 17, so move all references into the `Api17` wrapper
class. `GLES30` requires API 18.
Remove some defensive calls to check that a context is current. Checking for GL
errors after calls should catch the case of calling GL methods without a
context (which is also only likely to come up early in development), and these
checks were not done consistently across all methods anyway. That allows
removing the API 17 requirement from `SceneRenderer` (where previously it
wasn't documented).
PiperOrigin-RevId: 548669946
The streams return end-of-input if they read no samples, but know that
they are fully buffered to at least the clipped end time. This helps to
detect the end of stream even if there are no new buffers after the end
of the clip (e.g. for sparse metadata tracks).
The race condition occurs because the buffered position is evaluated
after reading the sample. So between reading "no sample" and checking
the buffered position, the source may have loaded arbitrary amounts
of data. This may lead to a situation where the source has not read
all samples, reads NOTHING_READ (because the queue is empty) and then
immediately returns end-of-stream (because the buffered position
jumped forward), causing all remaining samples in the stream to be
skipped. This can fixed by moving the buffered position check to
before reading the sample, so that it never exceeds the buffered
position at the time of reading "no sample".
#minor-release
PiperOrigin-RevId: 548646464
Also re-use the `CuesWithTimingSubtitle` implementation (previously a
private class inside `DelegatingSubtitleDecoder`) in `ExoPlayerCuesDecoder`.
PiperOrigin-RevId: 548612040
It will act similarly to `SubtitleDecoderFactory`, but return parsers instead of decoders. In turn `SubtitleDecoderFactory.createDecoder()` cab wrap those parsers with `DelegatingSubtitleDecoder`.
PiperOrigin-RevId: 548528054
`PgsDecoder` which used to be `SimpleSubtitleDecoder` will now be called `PgsParser` and implement `SubtitleParser` interface. For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `PgsParser` instance.
PiperOrigin-RevId: 548520549
This serves as a sort of test with more frames for the compositor for now
(before more varied video system tests come later when integrating with
Transformer), showing it doesn't error out and outputs the right
amount of frames.
Due to the VFPTestRunner having a 5s timeout, and mostly due to presubmit
emulators being very slow with OpenGL, there is a sort of limitation
to how many frames this type of test can have, depending on the test target
PiperOrigin-RevId: 548159762
`DvbDecoder` which used to be `SimpleSubtitleDecoder` will now be called `DvbParser` and implement `SubtitleParser` interface. There was, however, already a `DvbParser`, used by `DvbDecoder` behind the scenes. Hence, the refactoring only requires the existing `DvbParser` to adhere to the new `SubtitleParser` interface.
For backwards compatibility, we will have the same functionality as `DvbDecoder` provided by `DelegatingSubtitleDecoder` backed-up by a new `DvbParser` instance, available from the `SubtitleDecoderFactory`.
PiperOrigin-RevId: 548155759
`Tx3gDecoder` which used to be `SimpleSubtitleDecoder` will now be called `Tx3gParser` and implement `SubtitleParser` interface. For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `Tx3gParser` instance.
PiperOrigin-RevId: 548144492
Before this CL, the text with a scale span would wrap text, with because the scale wasn't taken into account when measuring the width of the overlay.
PiperOrigin-RevId: 548123626
Unlike ExoPlayer, when play/pause is called on MediaController, the playback suppression reason is set to default causing a change in it and subsequent callbacks on the Player.Listener.
PiperOrigin-RevId: 548088931
Add Ogg ID Header and Comment Header Pages to the Ogg encapsulated Opus for offload playback. This further matches the RFC 7845 spec and provides initialization data to decoders.
PiperOrigin-RevId: 548080222
These tests allow to verify the samples sent for decoding,
when decoders are reset, which samples are dropped as decode-only
for video and which samples are sent to the AudioTrack for playback.
The test exercises all combinations of merges where audio or video
is the primary track and where audio, video or both are clipped.
PiperOrigin-RevId: 548061254
These tests allow to verify the samples sent for decoding,
when decoders are reset, which samples are dropped as decode-only
for video and which samples are sent to the AudioTrack for playback.
The test exercises all combinations of clipping transitions for
sources that are either clipped at the start, the end, or both.
PiperOrigin-RevId: 547730824
When a renderer is pre-enabled (while another playback is still
ongoing), we pass mayRenderStartOfStream=false to Renderer.enable.
This ensures we don't show any first frames while the previous media
is still playing.
Currently, we never tell the renderer when we actually stop playing
the previous media so that it could render the start of the stream,
because we allow this as soon as the renderer is in STATE_STARTED and
we assume that we have to be in STATE_STARTED to make this stream
transition.
While this assumption is true, there are also cases where we can't
start the renderers because they are not ready yet and the video
renderer can't become ready because it didn't render its first frame.
This effectively blocks playback forever.
The most direct way of solving this, is to tell the renderer that
playback has transitioned and that it is now allowed to render the
start of the stream. This means it can never get blocked as described
above.
PiperOrigin-RevId: 547727347
SingleInputVideoGraph implements GraphInput now, so the asset loaders would
interface directly with SIVG, rather than VideoSampleExporter. This is to pave
way for multi-asset video processing.
PiperOrigin-RevId: 547561042
The setPlayerCommand and setSessionCommand builder methods contain assertions to ensure that at most one of these fields is set, but this left it possible to create a command button with no command by calling build on an empty builder.
PiperOrigin-RevId: 547488248
This change also marks the buttons of the custom layout as
enabled/disabled according to available commands in the controller.
Accordingly, `CommandButton.Builder.setEnabled(boolean)` is deprecated
because the value is overridden by the library.
Issue: androidx/media#38
#minor-release
PiperOrigin-RevId: 547272588
`Mp4WebvttDecoder` which used to be `SimpleSubtitleDecoder` will now be called `Mp4WebvttParser` and implement `SubtitleParser` interface. For backwards compatibility, we will have the same functionality provided by `DelegatingSubtitleDecoder` backed-up by a new `Mp4WebvttParser` instance.
PiperOrigin-RevId: 547248157
* Allow more than one input bitmap at a time.
* Allow Compositor to take in and set up an Executor. Otherwise,
Compositor resources may be created on one thread and accessed on another.
* Add a Compositor TestRunner to reuse test code more.
* Update VideoFrameProcessingTaskExecutor to use a new onError listener, so
that it's more reusable in non-DVFP contexts, like for Compositor.
PiperOrigin-RevId: 547206053
durationUs is almost always going to be a larger number than the sample
rate, so pass as the main value, rather than the multiplier.
PiperOrigin-RevId: 547193927
For sync-sample-only formats, we have an optimization to drop all buffers
with less than the start time when writing them to the queue.
For the same formats, if we set a new start time (=seek), we only seek
to the buffer at or before the start time. This means the first sample
in the queue is different depending on whether we seek to a start time
or set a start time and then write samples. This is inconsistent and
effectively means the first sample depends on a race condition between
the Loader thread (writing samples) and the playback thread (attempting
an initial seek in the already loaded samples).
The effect of this inconsistency is that we have to decode one sample
we don't need (and could have skipped) and that some tests become flaky
if the test setup runs into the mentioned race condition.
The fix is to change the SampleQueue seek method to also seek to
a sample at or after the specified time, to align the behavior to the
case where we write the same samples to an empty queue.
The change also clarifies the Javadoc of
MimeTypes.allSamplesAreSyncSamples to note that this should really only
return true if the samples have no "duration" that matters. Otherwise,
we could reasonably return true for most subtitle formats although it
would break subtitle display because we'd remove samples that start
before the seek time.
PiperOrigin-RevId: 547189941
This is needed for several subtitle formats where the `data` passed to
`SubtitleParser` doesn't contain any cue timing, such as PGS, CEA-608,
CEA-708, and DVB.
PiperOrigin-RevId: 547177127
