Renderers may be enabled for subsequent media items as soon as the current media item's renderer's isEnded() returns true. When a renderer is being enabled and the player is 'playing', that renderer is also started. When playing a mixed playlist of images and content with audio & video, the player may skip some image items because the early-starting of the audio renderer causes a clock update.
A solution is to only start the "early-enabled" renderers at the point of media transition and add a condition on DefaultMediaClock to use the standalone clock when reading-ahead and the renderer clock source is not in a started state.
Issue: androidx/media#1017
PiperOrigin-RevId: 613231227
This change aims to prioritise tracks that have a 'smooth enough for
video' frame rate, without always selecting the track with the highest
frame rate.
In particular MP4 files extracted from motion photos sometimes have two
HEVC tracks, with the higher-res one having a very low frame rate (not
intended for use in video playback). Before this change
`DefaultTrackSelector` would pick the low-fps, high-res track.
This change adds a somewhat arbitrary 10fps threshold for "smooth video
playback", meaning any tracks above this threshold are selected in
preference to tracks below it. Within the tracks above the threshold
other attributes are used to select the preferred track. We deliberately
don't pick the highest-fps track (over pixel count and bitrate), because
most users would prefer to see a 30fps 4k track over a 60fps 720p track.
This change also includes a test MP4 file, extracted from the existing
`jpeg/pixel-motion-photo-2-hevc-tracks.jpg` file by logging
`mp4StartPosition` in
[`MotionPhotoDescription.getMotionPhotoMetadata`](b930b40a16/libraries/extractor/src/main/java/androidx/media3/extractor/jpeg/MotionPhotoDescription.java (L123))
and then using `dd`:
```
mp4StartPosition=2603594
$ dd if=jpeg/pixel-motion-photo-2-hevc-tracks.jpg \
of=mp4/pixel-motion-photo-2-hevc-tracks.mp4 \
bs=1 \
skip=2603594
```
----
This solution is in addition to the `JpegMotionPhotoExtractor` change
made specifically for these two-track motion photos in
5266c71b3a.
We will keep both changes, even though that change is not strictly
needed after this one, because adding the role flags helps to
communicate more clearly the intended usage of these tracks. This
change to consider FPS seems like a generally useful improvement to
`DefaultTrackSelector`, since it seems unlikely we would prefer a 5fps
video track over a 30fps one.
Issue: androidx/media#1051
PiperOrigin-RevId: 611015459
These audio offload failure recovery tests model the DefaultAudioSink failing at audio track init and write operations in offload mode. Playback should recover and try again as DefaultAudioSink will disable offload mode.
PiperOrigin-RevId: 610372935
The new test introduced in 45bd5c6f0a is flaky because we only
wait until the media is fully buffered. However, we can't fully
control how much of this data is initially read by the Robolectric
codec and thus the output dump files (containing these codec
interactions) are flaky.
This can be fixed by fully playing the media once and then seeking
back instead.
#minor-release
PiperOrigin-RevId: 603324068
If seeking between last image sample and end of the file where the current stream is not final, then EoS sample will not be provided to `ImageRenderer`. ImageRenderer must still produce the last image sample.
PiperOrigin-RevId: 603312090
This file is CBR encoded with LAME, so it has an `Info` header (the CBR
equivalent to `Xing`).
A follow-up change will use this file in `Mp3ExtractorTest`.
Issue: androidx/media#878
PiperOrigin-RevId: 595938327
HLS distinguishes between 'subtitles' (WebVTT or TTML distributed in
separate files with their own playlist) and 'captions' (CEA-608 or 708,
distributed muxed into the video file).
The format transformation added in 7b762642db
only applies to subtitles and not captions. This change makes the same
transformation for caption formats.
This resolves an error like:
```
SampleQueueMappingException: Unable to bind a sample queue to TrackGroup with MIME type application/cea-608.
```
Also add two playback tests for HLS CEA-608, one that parses during
decoding (old way) and one during extraction (new way). Adding these
tests is what alerted me to this issue.
PiperOrigin-RevId: 592571284
This was generated by combining the existing `ts/bbb_2500ms.ts` test
asset and a temporary `.srt` file using
https://cloud.google.com/transcoder/docs/how-to/captions-and-subtitles
This doesn't directly reproduce the problem fixed by
7ca26f898d,
because the CEA-608 subs are structured differently to the stream I
discovered the problem with (from Issue: androidx/media#887). However this test
does fail if that fix is reverted after
486230fbd7.
I'm also not able to repro the character duplication reported in
Issue: androidx/media#887 by just changing the manifest in this CL. I'm not yet
sure on the exact differences between the stream provided on GitHub
and this stream.
This stream does provide some regression protection, because it
currently fails with 'new' subtitle parsing
(`DashMediaSource.Factory.experimentalParseSubtitlesDuringExtraction(true)`),
though I'm not sure on the exact reason for that yet.
PiperOrigin-RevId: 592476328
The MP4 data in JPEG motion photos can contain multiple `video/hevc` tracks, but only the first is at a playable frame rate while the others are low-fps, high-res tracks designed for specific use-cases (not direct video playback).
ExoPlayer currently selects the unplayable track by default, because it
has a higher resolution. This change introduces a flag to
`Mp4Extractor` that results in the first video track being marked as
`ROLE_FLAG_MAIN`, and all subsequent video tracks `ROLE_FLAG_ALTERNATE`
- this then results in the playable lower-res track being selected by
default.
PiperOrigin-RevId: 589832072
This image has two video tracks in the MP4 data, one is a 'real' video
which we want to play by default, and the other is a low-fps video track
which isn't intended to be directly played, it's encoded in HEVC for
compression and decoding efficiency.
This test demonstrates ExoPlayer's current behaviour default extraction
and playback, which results in selecting the high-res, low-fps track
(actually single sample in this example), instead of playing the actual
video.
PiperOrigin-RevId: 588068908
As Opus decoders skip some bytes prior to playback during a seek, the renderer for bypass playback should send samples to the decoder even if they would be decode-only. However, the renderer should not send samples with time preceding that range. This change adds that constraint.
#minor-release
PiperOrigin-RevId: 588014983
This removes the flakiness of the HLS playback test as well.
Previously, this flow only worked for standalone WebVTT subtitles (7b762642db)
PiperOrigin-RevId: 582359383
The test is hidden behind the `@Ignore` annotation due to some
flakiness. However, it will be removed when the subtitle parsing is
moved to a pre-sample-queue architecture.
The test media was created with:
```shell
$ cp ../../dash/standalone-ttml/sample.xml sample.ttml
$ MP4Box -add sample.ttml sample.text.mp4
$ MP4Box -frag 10000 sample.text.mp4
$ rm sample.ttml
```
PiperOrigin-RevId: 582347113
This change applies to standalone WebVTT files linked directly from the manifest.
Since DASH only supports stand-alone IMSC1 (TTML) and WebVTT text files, this change concludes the support extension of text-based subtitle files to be parse during extraction.
PiperOrigin-RevId: 577468830
As Opus decoders skip some bytes prior to playback during a seek, the renderer for bypass playback should send samples to the decoder even if they would be decode-only.
#minor-release
PiperOrigin-RevId: 574494666
This currently only applies to subtitles muxed into mp4 segments, and
not standalone text files linked directly from the manifest.
Issue: androidx/media#288
#minor-release
PiperOrigin-RevId: 572263764
The existing `Subtitle` handling code is left intact to support the
legacy post-`SampleQueue` decoding path for now.
This also includes full support for merging overlapping `CuesWithTiming`
instances, which explains the test dump file changes, and which should
resolve the following issues (if used with the
decoder-before-`SampleQueue` subtitle logic added in
5d453fcf37):
* Issue: google/ExoPlayer#10295
* Issue: google/ExoPlayer#4794
It should also help resolve Issue: androidx/media#288, but that will also require
some changes in the DASH module to enable pre-`SampleQueue` subtitle
parsing (which should happen soon).
#minor-release
PiperOrigin-RevId: 571021417
The test is hidden behind the Ignore annotation due to some flakiness just like `webvttInMp4`. However, it will be removed when the subtitle parsing is moved to a pre-sample-queue architecture.
#minor-release
PiperOrigin-RevId: 570376275
This is equivalent to the existing `scaleLargeTimestamp` method with the
following changes/improvements:
* No longer specific to timestamps (there was nothing inherently
time-specific about the logic in `scaleLargeTimestamp`, but the name
and docs suggested it shouldn't be used for non-timestamp use-cases).
* Additional 'perfect division' checks between `value` and `divisor`.
* The caller can now provide a `RoundingMode`.
* Robust against `multiplier == 0`.
* Some extra branches before falling through to (potentially lossy)
floating-point math, including trying to simplify the fraction with
greatest common divisor to reduce the chance of overflowing `long`.
This was discussed during review of 6e91f0d4c5
This change also includes some golden test file updates - these
represent a bug fix where floating-point maths had previously resulted
in a timestamp being incorrectly rounded down to the previous
microsecond. These changes are due to the 'some more branches' mentioned
above.
PiperOrigin-RevId: 564760748
This change makes capturing components in CapturingRenderersFactory
to dump data only they have previously captured any. This is so we can
extend the CapturingRenderersFactory with more renderers that do not
capture data in pre-existing tests and we don't have to change the
golden files.
PiperOrigin-RevId: 561973645
also improves tests to hash bitmap now that we know the bitmaps are decoded in roboelectic tests,
as well as some minor fixes in the DefaultImageDecoder
PiperOrigin-RevId: 561946073
This is the same change as 8655429af7, just on the second call site
of readDiscontinuity. The tests didn't cover this case yet because
they never queued more than 2 items in a playlist.
PiperOrigin-RevId: 553485244
It is not part of the `SubtitleParser` interface's promise to give events for all changes to do with `Cue` objects. So while in the past, we would have `endTimeUs` of one Cue event being the `startTimeUs` of the next one, now we have that encoded in `durationUs` and can skip event with empty `Cues`.
PiperOrigin-RevId: 549629157
This is no longer flaky because WebVTT subtitles are decoded on the
loading thread (enabled in this test with
`defaultExractorsFactory.setTextTrackTranscodingEnabled(true)` - and
supported for WebVTT since f0f24aa0d4).
PiperOrigin-RevId: 549594291
The end-to-end test output for the overlapping SRT and SSA subtitles
is currently incorrect. They will be fixed in a future change that
updates `TextRenderer` to support this overlap.
The 'extra' samples visible in the extractor test output files are
'empty cue list' samples produced by `SsaParser`. They will go away
when this implementation is updated to remove this behaviour and rely
on `CuesWithTiming.durationUs` instead (the 'empty list' behaviour is
not required by the `SubtitleParser` interface).
PiperOrigin-RevId: 549264593
*** Original commit ***
Rollback of b69b33206e
*** Original commit ***
Mark output sample as decode-only based on start time
We currently do the same check on the input timestamps and
expect the output timestamps to match. Some codecs produce
samples with modified timestamps and the logic is a lot safer
when the comparison with the start time is done on the output
side of the codec.
Issue: google/ExoPlayer#11000
***
***
PiperOrigin-RevId: 549019403
This was done because it was deemed correct to only start at timestamp
zero when the code was originally written. However, in case of
prerolling from a keyframe, many samples will get the same timestamp,
which is not correct and interferes with downstream logic
that deals with timestamps.
PiperOrigin-RevId: 548986160
This is a step towards adding general support for overlapping
subtitles in these formats (and others), both muxed and sideloaded:
* Issue: google/ExoPlayer#10295
* Issue: google/ExoPlayer#4794
This change adds these files to the end-to-end playback tests too, but
the subtitle track is currently disabled because renderer-side subtitle
parsing causes flaky tests (due to an uncontrolled thread in
`SimpleSubtitleDecoder`). The subtitle track will be re-enabled in
a follow-up change when loading-side subtitle parsing is added (so the
tests will no longer be flaky). At this point the overlapping subtitles
**still** won't be supported end-to-end, but a second change will
resolve this will changes in `TextRenderer` - which will change the
end-to-end playback dumps to reflect the overlapping subtitles.
PiperOrigin-RevId: 548705032
Add Ogg ID Header and Comment Header Pages to the Ogg encapsulated Opus for offload playback. This further matches the RFC 7845 spec and provides initialization data to decoders.
PiperOrigin-RevId: 548080222
These tests allow to verify the samples sent for decoding,
when decoders are reset, which samples are dropped as decode-only
for video and which samples are sent to the AudioTrack for playback.
The test exercises all combinations of merges where audio or video
is the primary track and where audio, video or both are clipped.
PiperOrigin-RevId: 548061254
These tests allow to verify the samples sent for decoding,
when decoders are reset, which samples are dropped as decode-only
for video and which samples are sent to the AudioTrack for playback.
The test exercises all combinations of clipping transitions for
sources that are either clipped at the start, the end, or both.
PiperOrigin-RevId: 547730824
We currently only log the input buffers to the codec, but the test
will be more complete and provide more insights into the playback
behavior if we also dump more information about these input buffers
(timestamps and flags) and also all output buffers with their
timestamps and whether they are rendererd (only relevant for video).
And also use the CapturingAudioSink to output the audio data we produce.
PiperOrigin-RevId: 546898937
Some adjustments to the test to make it more correct and prevent it
from becoming flaky:
- Use separate output dump files per test setup. Once we add more data
to these files, they are not guaranteed to be the same anymore.
- Use a seek position that is actually behind the midroll as described
in the test setup.
- Change ad insertion position to ensure the ad group duration doesn't
exceed the underlying media duration.
- Add a wait for isLoading to ensure the late insertion of an ad group
happens consistently at the same processing stage.
PiperOrigin-RevId: 546825183
