WebvttExtractor will no longer be wrapped in SubtitleTranscodingExtractor, but instead use SubtitleTranscodingExtractorOutput under the hood.
A new constructor will take a boolean parameter to toggle between subtitle parsing during extraction (before the sample queue) or during decoding (after the sample queue).
PiperOrigin-RevId: 597604942
An audio file can only play sound between two PCM samples (the 'start'
and 'end' of section of a wave form). Therefore when calculating
duration from a count of PCM samples we need to subtract one first (the
'end' sample which has no duration of its own).
This only changes durations by one PCM sample (21us - 22us for 44.1kHz sample
rate).
PiperOrigin-RevId: 596990306
The extractor knows the PCM encoding of the losslessly
encoded data in the samples and should set it in the
Format to allow downstream components to use this information.
PiperOrigin-RevId: 596974863
This file is CBR encoded with LAME, so it has an `Info` header (the CBR
equivalent to `Xing`).
A follow-up change will use this file in `Mp3ExtractorTest`.
Issue: androidx/media#878
PiperOrigin-RevId: 595938327
Changes includes;
1. Public API to enable fMP4 and to pass fragment duration.
2. Added `FragmentedMp4Writer`.
3. Added logic to create fragments based on given fragment duration.
4. Write "moov" box only once in the beginning.
3. Add all the required boxes for current implementation.
4. Unit tests for all the new boxes.
5. E2E test for generating fMP4.
Note: The output file is un seek-able with this first implementation.
PiperOrigin-RevId: 594426486
Mp4Muxer does not support out of order B-frames. Currently it
silently writes out of order B-frames, producing an invalid file (with
negative sample durations).
Although `Mp4Extractor` is somehow able to process this invalid file and
`Exoplayer` is able to play it but that is unexpected.
The `sample.mp4` test file contains B frames. Other test files does not
contain `H264 video + AAC audio` format hence created a new test file by
running `sample.mp4` via `Transformer` after applying some effects.
PiperOrigin-RevId: 594016144
HLS distinguishes between 'subtitles' (WebVTT or TTML distributed in
separate files with their own playlist) and 'captions' (CEA-608 or 708,
distributed muxed into the video file).
The format transformation added in 7b762642db
only applies to subtitles and not captions. This change makes the same
transformation for caption formats.
This resolves an error like:
```
SampleQueueMappingException: Unable to bind a sample queue to TrackGroup with MIME type application/cea-608.
```
Also add two playback tests for HLS CEA-608, one that parses during
decoding (old way) and one during extraction (new way). Adding these
tests is what alerted me to this issue.
PiperOrigin-RevId: 592571284
This was generated by combining the existing `ts/bbb_2500ms.ts` test
asset and a temporary `.srt` file using
https://cloud.google.com/transcoder/docs/how-to/captions-and-subtitles
This doesn't directly reproduce the problem fixed by
7ca26f898d,
because the CEA-608 subs are structured differently to the stream I
discovered the problem with (from Issue: androidx/media#887). However this test
does fail if that fix is reverted after
486230fbd7.
I'm also not able to repro the character duplication reported in
Issue: androidx/media#887 by just changing the manifest in this CL. I'm not yet
sure on the exact differences between the stream provided on GitHub
and this stream.
This stream does provide some regression protection, because it
currently fails with 'new' subtitle parsing
(`DashMediaSource.Factory.experimentalParseSubtitlesDuringExtraction(true)`),
though I'm not sure on the exact reason for that yet.
PiperOrigin-RevId: 592476328
This is because currently
1. Player sets a surfaceView to render to
2. Player intializes the renderer
3. MCVR initializes the VideoSinkProvider, by extension VideoGraph
But when 1 happens, MCVR doesn't set the surfaceView on the VideoGraph because
it's not initialized. Consequently after VideoGraph is initialized, it doesn't
have a surface to render to, and thus dropping the first a few frames.
Also adds a test for first frame to verify the correct first frame is rendered.
PiperOrigin-RevId: 591228174
The MP4 data in JPEG motion photos can contain multiple `video/hevc` tracks, but only the first is at a playable frame rate while the others are low-fps, high-res tracks designed for specific use-cases (not direct video playback).
ExoPlayer currently selects the unplayable track by default, because it
has a higher resolution. This change introduces a flag to
`Mp4Extractor` that results in the first video track being marked as
`ROLE_FLAG_MAIN`, and all subsequent video tracks `ROLE_FLAG_ALTERNATE`
- this then results in the playable lower-res track being selected by
default.
PiperOrigin-RevId: 589832072
This image has two video tracks in the MP4 data, one is a 'real' video
which we want to play by default, and the other is a low-fps video track
which isn't intended to be directly played, it's encoded in HEVC for
compression and decoding efficiency.
This test demonstrates ExoPlayer's current behaviour default extraction
and playback, which results in selecting the high-res, low-fps track
(actually single sample in this example), instead of playing the actual
video.
PiperOrigin-RevId: 588068908
As Opus decoders skip some bytes prior to playback during a seek, the renderer for bypass playback should send samples to the decoder even if they would be decode-only. However, the renderer should not send samples with time preceding that range. This change adds that constraint.
#minor-release
PiperOrigin-RevId: 588014983
Both the extension OPUS decoder and the OMX/C2 MediaCodec
implementations for OPUS and VORBIS decode into the channel
layout defined by VORBIS. See
https://www.xiph.org/vorbis/doc/Vorbis_I_spec.html#x1-140001.2.3
While this is technically correct for a stand-alone OPUS or VORBIS
decoder, it doesn't match the channel layout expected by Android.
See https://developer.android.com/reference/android/media/AudioFormat#channelMask
The fix is to apply the channel mapping after decoding if needed.
Also add e2e tests with audio dumps for the extension renderer,
including a new 5.1 channel test file.
Issue: google/ExoPlayer#8396
#minor-release
PiperOrigin-RevId: 588004832
Before, a translucent overlay over an opaque video would result in a
translucent output. This is not consistent with physical properties of light
(if putting a translucent object in front of an opaque object, you can't see
behind the opaque object).
Using the mixing properties from DefaultVideoCompositor.
PiperOrigin-RevId: 586636275
`fromBundle` doesn't distinguish between `FIELD_BITMAP` and `FIELD_TEXT`
being present with a null value, or being absent, so we might as well
avoid including them when the value is null.
I've separated this from a later change to add
`Cue.toSerializableBundle` which will also skip setting a bitmap value
into the `Bundle` if `this.bitmap == null`. This is partly because it
results in changes to a lot of extractor test dump files, and it's
easier to review that as a separate change.
PiperOrigin-RevId: 586626141
This change fixes a bug with seeking forward in MIDI. When seeking forward,
the progressive media period attempts to seek within the sample queue, if a
key-frame exists before the seeking position. With MIDI, however, we can
only skip Note-On and Note-Off samples and all other samples must be sent
to the MIDI decoder.
When seeking outside the sample queue, the MidiExtractor already
instructs the player to start from the beginning of the MIDI input. With
this change, only the first output sample is a key-frame, thus the
progressive media period can no longer seek within the sample queue and
is forced to seek from the MIDI input start always.
Issue: androidx/media#704
#minor-release
PiperOrigin-RevId: 584321443
When transmuxing, the `EncodedSampleExporter` maintains a queue of input
buffers that get filled with encoded data by the asset loader. The number of
buffers was limited to avoid using more and more memory if producer (asset
loader) gets far ahead of the consumer (exporter).
Previously this limit was fixed at 10 buffers, but increasing the number of
buffers can make some transmux operations much faster. Allow allocating between
a min and max number of buffers, and also set a target allocation size beyond
which new buffers can't be allocated. This allows audio formats which require
many small buffers to be processed more quickly, while preventing allocating
too much memory for hypothetical very high bitrate formats.
'Remove video' edits on local videos in particular get much faster, because
audio buffers are very short and there are lots of them. With a sample 10
minute video, a 'remove video' edit took 2 seconds (36 seconds before this
change). With a sample 1 minute removing video took 0.25 seconds after this
change (2.5 seconds before).
The speed improvement is smaller for other types of edits that retain the video
track. Transmuxing a 10 minute video retaining the video track took 26 seconds
(40 seconds before).
PiperOrigin-RevId: 583390284
This removes the flakiness of the HLS playback test as well.
Previously, this flow only worked for standalone WebVTT subtitles (7b762642db)
PiperOrigin-RevId: 582359383
The test is hidden behind the `@Ignore` annotation due to some
flakiness. However, it will be removed when the subtitle parsing is
moved to a pre-sample-queue architecture.
The test media was created with:
```shell
$ cp ../../dash/standalone-ttml/sample.xml sample.ttml
$ MP4Box -add sample.ttml sample.text.mp4
$ MP4Box -frag 10000 sample.text.mp4
$ rm sample.ttml
```
PiperOrigin-RevId: 582347113
Use renderengine's PQ to SDR tone-mapping implementation instead
of naive implementation from before.
This improves luminance on highlights, as seen in the test image.
PiperOrigin-RevId: 582318045
The dump file diff as part of this change is because using AudioGraph
means the 2nd item is automatically edited to match the AudioFormat of
the 1st item {44.1KHz mono}, rather than {48KHz stereo}.
Manually verified that for the 2nd item, data output:
* Before: 66936 bytes (16734 frames) output = 348_625us of audio.
* After: 30750 bytes (15375 frames) output = 348_639us of audio.
The small final buffer is caused by SonicAudioProcessor outputting all
pending data when EOS queued, and is WAI.
PiperOrigin-RevId: 580494578
This change applies to standalone WebVTT files linked directly from the manifest.
Since DASH only supports stand-alone IMSC1 (TTML) and WebVTT text files, this change concludes the support extension of text-based subtitle files to be parse during extraction.
PiperOrigin-RevId: 577468830
As Opus decoders skip some bytes prior to playback during a seek, the renderer for bypass playback should send samples to the decoder even if they would be decode-only.
#minor-release
PiperOrigin-RevId: 574494666
If the `Subtitle` has 'active' cues at `OutputOptions.startTimeUs`, this
change ensures these are emitted in a `CuesWithTiming` with
`CuesWithTiming.startTimeUs = OutputOptions.startTimeUs`. If
`OutputOptions.outputAllCues` is also set, then another `CuesWithTiming`
is emitted at the end that covers the 'first part' of the active cues,
and ends at `OutputOptions.startTimeUs`.
As well as adding some more tests to `LegacySubtitleUtilWebvttTest`,
this change also adds more tests for `TtmlParser` handling of
`OutputOptions`, which transitively tests the behaviour of
`LegacySubtitleUtil`.
#minor-release
PiperOrigin-RevId: 573151016
This currently only applies to subtitles muxed into mp4 segments, and
not standalone text files linked directly from the manifest.
Issue: androidx/media#288
#minor-release
PiperOrigin-RevId: 572263764
The flag is no longer used by our components and only set and checked
in a few places to guarantee compatiblity with existing renderers and
decoders that still use it.
The flag will be removed in the future due to its design limitations.
#minor-release
PiperOrigin-RevId: 571291168
The existing `Subtitle` handling code is left intact to support the
legacy post-`SampleQueue` decoding path for now.
This also includes full support for merging overlapping `CuesWithTiming`
instances, which explains the test dump file changes, and which should
resolve the following issues (if used with the
decoder-before-`SampleQueue` subtitle logic added in
5d453fcf37):
* Issue: google/ExoPlayer#10295
* Issue: google/ExoPlayer#4794
It should also help resolve Issue: androidx/media#288, but that will also require
some changes in the DASH module to enable pre-`SampleQueue` subtitle
parsing (which should happen soon).
#minor-release
PiperOrigin-RevId: 571021417
The test is hidden behind the Ignore annotation due to some flakiness just like `webvttInMp4`. However, it will be removed when the subtitle parsing is moved to a pre-sample-queue architecture.
#minor-release
PiperOrigin-RevId: 570376275
Move remote 8K file to local and trim to 320ms.
Trim done with ffmpeg:
`ffmpeg -i {remote_file} -t 0.3 -c:v copy -c:a copy 8k24fps_300ms.mp4`
PiperOrigin-RevId: 569449962
This is equivalent to the existing `scaleLargeTimestamp` method with the
following changes/improvements:
* No longer specific to timestamps (there was nothing inherently
time-specific about the logic in `scaleLargeTimestamp`, but the name
and docs suggested it shouldn't be used for non-timestamp use-cases).
* Additional 'perfect division' checks between `value` and `divisor`.
* The caller can now provide a `RoundingMode`.
* Robust against `multiplier == 0`.
* Some extra branches before falling through to (potentially lossy)
floating-point math, including trying to simplify the fraction with
greatest common divisor to reduce the chance of overflowing `long`.
This was discussed during review of 6e91f0d4c5
This change also includes some golden test file updates - these
represent a bug fix where floating-point maths had previously resulted
in a timestamp being incorrectly rounded down to the previous
microsecond. These changes are due to the 'some more branches' mentioned
above.
PiperOrigin-RevId: 564760748
Modifying dumping to not required "released" to be called.
Track index is an arbitrary value based on the order of addTrack calls.
Samples are dumped by track (rather than as soon as they are written),
so it's preferable to use a value that provides more context.
By using the track type as a key, dump files will be more deterministic
and will have more similarities when branched.
PiperOrigin-RevId: 563700982
This change makes capturing components in CapturingRenderersFactory
to dump data only they have previously captured any. This is so we can
extend the CapturingRenderersFactory with more renderers that do not
capture data in pre-existing tests and we don't have to change the
golden files.
PiperOrigin-RevId: 561973645
also improves tests to hash bitmap now that we know the bitmaps are decoded in roboelectic tests,
as well as some minor fixes in the DefaultImageDecoder
PiperOrigin-RevId: 561946073
Implement VideoCompositor support of:
* Different input and output sizes
* CompositorSettings, to customize output size based on input texture sizes
* OverlaySettings, to place an input frame in an arbitrary position on
the output frame.
Also, refactor Overlay's matrix logic to make it more reusable between
Compositor and Overlays
PiperOrigin-RevId: 561931854
This file decompresses to 3024*4032*4 bytes ~= 48
MB, which leads to OOM errors on some devices.
Use a lower resolution test image for tests, to
still test a portrait image but without hitting
OOM errors.
PiperOrigin-RevId: 557861063
In addition to this being how Alpha should be handled, tests do fail under
API 29 without this CL. Unfortunately while these tests do fail under API 29
without this CL, we currently only run this test on API 33 emulators, so we
didn't catch this failure earlier (until compositor tests on mh failed on all
devices under API 29 and succeeded on all at or over 29).
PiperOrigin-RevId: 557781757
This test uses an existing CBR test asset, and shows that (as expected)
`isSeekable = true` even in the `unknown_length` dump file.
PiperOrigin-RevId: 557455364
This effect can scale Alpha values.
This is needed for testing Compositor, and coincidentally was requested
by a partner in the past. This also will generally be more useful in
full multi-asset, where we may want to have one input have different alpha
values than another
Also, update fragment_shader_copy_es2.glsl to not throw away alpha values.
This means ThumbnailStripShaderProgram (the only place this glsl is used) will
consider input alpha and output alpha appropriately
PiperOrigin-RevId: 555915092
To ensure that for each output bitmap from the compostor, the right input
timestamps were used.
Only applied on a subset of tests to avoid needing to upload+maintain
too many files/size in the test binary, especially when it would test
duplicate behavior
PiperOrigin-RevId: 555222530
The test file decompresses to about 48 MB, which can cause an
`OutOfMemoryError` on lower end devices.
Also, a test appears to apply a rotation to this bitmap without downscaling
first, leading to exceeding a maximum texture width/height of 4096 on some
devices.
Replace the test file with one that has a quarter of the original's
width/height so that these tests can pass reliably even on lower end
devices.
PiperOrigin-RevId: 554507185
