This allows us to also plumb whether we're doing MediaCodec tone-mapping,
which will be used in a follow-up CL in the ExoAssetLoaderVideoRenderer.
PiperOrigin-RevId: 601774435
Allow setAdditionalRotationDegrees to be called with same rotation after tracks added. This is needed for processes that mux files partially trim optimization so they don't error out after hitting the check state
Manually tested to ensure trim optimization succeeds, automated test added here as well
PiperOrigin-RevId: 601081778
This file is CBR encoded with LAME, so it has an `Info` header (the CBR
equivalent to `Xing`).
A follow-up change will use this file in `Mp3ExtractorTest`.
Issue: androidx/media#878
PiperOrigin-RevId: 595938327
Mp4Muxer does not support out of order B-frames. Currently it
silently writes out of order B-frames, producing an invalid file (with
negative sample durations).
Although `Mp4Extractor` is somehow able to process this invalid file and
`Exoplayer` is able to play it but that is unexpected.
The `sample.mp4` test file contains B frames. Other test files does not
contain `H264 video + AAC audio` format hence created a new test file by
running `sample.mp4` via `Transformer` after applying some effects.
PiperOrigin-RevId: 594016144
Also adds an alternate way to configure the AudioGraph.
Apps should no longer need to ensure that inputs have the same sample
rate.
PiperOrigin-RevId: 588747431
Added a new ABANDONED option so one can tell the difference between when the optimization has been requested but not applied vs not requested at all. also changed the ordering do better represent the hierarchy of failure modes
PiperOrigin-RevId: 588720513
* @Before and @After usage. [1].
* static fields before non-static. [2].
@Before method should typically be paired with an @After method,
focused on ensuring that the component is released regardless of what
the test does.
In tests, inlining final class variables is preferrable [1]. In general
things like the file path should be part of the test (the @Rule means
we don't need before/after) if only used once.
Statically importing values and using them directly is preferable to
having a variable declared as non-final that's effectively final,
because from a readability perspective someone can see (the caps) that
the value is final static and immutable, so doesn't have to check if
it's changed/reassigned.
PiperOrigin-RevId: 586697887
Using `Integer.MAX_VALUE` risks causing arithmetic overflow in the codec
implementation.
Issue: androidx/media#810
#minor-release
PiperOrigin-RevId: 585104621
When transmuxing, the `EncodedSampleExporter` maintains a queue of input
buffers that get filled with encoded data by the asset loader. The number of
buffers was limited to avoid using more and more memory if producer (asset
loader) gets far ahead of the consumer (exporter).
Previously this limit was fixed at 10 buffers, but increasing the number of
buffers can make some transmux operations much faster. Allow allocating between
a min and max number of buffers, and also set a target allocation size beyond
which new buffers can't be allocated. This allows audio formats which require
many small buffers to be processed more quickly, while preventing allocating
too much memory for hypothetical very high bitrate formats.
'Remove video' edits on local videos in particular get much faster, because
audio buffers are very short and there are lots of them. With a sample 10
minute video, a 'remove video' edit took 2 seconds (36 seconds before this
change). With a sample 1 minute removing video took 0.25 seconds after this
change (2.5 seconds before).
The speed improvement is smaller for other types of edits that retain the video
track. Transmuxing a 10 minute video retaining the video track took 26 seconds
(40 seconds before).
PiperOrigin-RevId: 583390284
For pause and resume feature we need to use same `MuxerWrapper`
for `remuxing processed video` and then to `process remaining video`.
In order to use same `MuxerWrapper` across `different Exports`
we need to preserve its state.
PiperOrigin-RevId: 564728396
Modifying dumping to not required "released" to be called.
Track index is an arbitrary value based on the order of addTrack calls.
Samples are dumped by track (rather than as soon as they are written),
so it's preferable to use a value that provides more context.
By using the track type as a key, dump files will be more deterministic
and will have more similarities when branched.
PiperOrigin-RevId: 563700982
Switch to SparseArray<List> rather than a Map<Integer, List>.
Track indices now determine sample list dumping, reducing flakiness.
PiperOrigin-RevId: 561658586
For pause and resume feature we need to find the timestamp of
the last sync sample in an MP4 file. The Mp4ExtractorWrapper
provides an easy to use API on top of Mp4Extractor.
PiperOrigin-RevId: 560113112
Covers all permutations&combinations of:
* AV asset.
* AV asset with effects.
* AV asset with silence.
* AV asset with silence and effects.
Note that the video is not relevant (therefore transmuxed), but is
needed for silence generation.
PiperOrigin-RevId: 558734593
Rename AudioMixerImpl to DefaultAudioMixer.
Removes the AudioMixerImpl specific getOutputAudioFormat, as the caller
defines and sets this.
PiperOrigin-RevId: 555887722
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.
Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.
Regression tests reproduce failure without fix, but pass with it.
PiperOrigin-RevId: 550853714
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.
Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.
PiperOrigin-RevId: 548994584
On a MediaItem change, the input Format (and Effects to apply) may be
different. Therefore the AudioProcessingPipeline must be reconfigured
to determine what processing is active, and what the AudioFormat of the
data output is. In the event that it is different, additional
AudioProcessor instances must be used to ensure the encoder will still
be able to accept the audio buffers.
PiperOrigin-RevId: 544338451
Goal of tests (SequenceExportTest) that use this media is for the
silence and the media to match exactly with audio format, however
`sample_with_increasing_timestamps.mp4` had a different sample rate.
testvid_1022ms.mp4: channel count = 2, sample rate = 44100.
PiperOrigin-RevId: 543458948
With the upcoming "handle format changes" CL, stereo -> mono audio
would add an AudioProcessor. Robolectric decodes output encoded data,
which crashes some AudioProcessors because the number of frames may not
be an integer.
PiperOrigin-RevId: 542568875
Audio only tests are now using RAW audio where possible, which is
passed through the Robolectric decoders/encoders, and can be handled by
the AudioProcessor instances accurately.
PiperOrigin-RevId: 541648853
Issue: When running the Transformer related test cases, the tests are flaky
because the order in which audio and video samples are interleaved seems to
differ in few instances.
Root cause: When running a transformation the sample producer (Asset loader)
and sample consumer (Sample pipeline) both runs on different thread and
theoretically there is no reason for behaviour to be deterministic because
the number of samples produced/written depends on how fast individual thread
works. So it is indeed surprising that test somehow worked deterministically in
majority of instances (may be something to do with Robolectric environment).
Solution: Since we don't expect the order of sample interleaving to be deterministic, make the dumping logic deterministic where all the video
samples will be collected and then dumped together (similarly for audio). This would mean we won't be able to see the interleaving so for that we need to
add separate test case verifying the interleaving logic only.
Pending: Test case for interleaving.
PiperOrigin-RevId: 540930871
The existing NullableType has been deprecated 5 years ago and causes
crashes in Kotlin apps because Kotlin doesn't recognize this annotation
as a nullable type annotation.
While we can't align on a single @Nullable annotation yet, we can at
least replace this one by JSR305's @Nonnull(MAYBE) as it fulfils all
requirements, including full Kotlin compatiblity. To avoid the
cumbersome name, we can redefine it as our own @NullableType
annotation. (We can't use @Nullable to avoid name clashes with the main
@Nullable annotation from AndroidX)
Issue: google/ExoPlayer#6792
PiperOrigin-RevId: 540497469
Mp4Muxer already supports writing Mp4LocationData so added that
as supported Metadata entry.
Support for more Metadata entries will be added in upcoming CLs.
PiperOrigin-RevId: 534473866
Previously `ChannelMixingAudioProcessor` output float because it was
implemented using the audio mixer's float mixing support.
Move the implementation over to just using the `ChannelMixingMatrix` and make
it publicly visible in the common module so it can be used by apps for both
playback and export.
Also resolve a TODO that no longer had a bug attached by implementing support
for putting multiple mixing matrices to handle different input audio channel
counts, and fix some nits in the test code.
Tested via unit tests and manually configuring a `ChannelMixingAudioProcessor`
in the transformer demo app and playing an audio stream that identifies
channels, and verifying that they are remapped as expected.
PiperOrigin-RevId: 523653901
Simplify the audio encoder input timestamp calculation. The new calculation
avoids drifting by tracking the total number of bytes encoded rather than
tracking the timestamp and remainder separately, and also makes the timestamps
match the decoder output buffer timestamps.
Also switch one of the export tests that was passing through AMR samples over
to using WAVE audio. The problem with using AMR is that the compressed samples
are not necessarily an integer number of audio frames and the shadow decoder
would pass them from input to output, so the audio encoder was receiving
non-integer numbers of audio frames.
Tested by logging the timestamps at the decoder output and encoder input with
forcing transcoding audio, and verifying that after this change the audio
timestamps are no longer off by one.
PiperOrigin-RevId: 523409869
Handling of the stream offset and start position was unnecessarily
complex and even incorrect. It was going to be an issue for
concatenation of video and image input.
The stream offset is the offset added before decoding/encoding to
make sure it doesn’t fail in case of negative timestamps (which do
rarely occur).
The start position is equal to the stream offset, plus the clipping
start time if the media is clipped.
Before this change:
- Samples were offset by the start position before decoding, and this
offset was removed before muxing.
- The startPosition of the first MediaItem in a sequence was used for
all the MediaItems in this sequence (which is incorrect).
- The stream offset was removed before applying the GL effects and
added back before encoding so that it was not visible to the OpenGL
processing.
After this change:
- The start position is subtracted in the AssetLoader, so that the
downstream components don’t have to deal with the stream offsets and
start positions.
- Decoded samples with negative timestamps are not passed to the
SamplePipelines. The MediaMuxer doesn’t handle negative timestamps
well. If a stream is 10 secondes long and starts at timestamp -2
seconds, the output will only contain the samples corresponding to the
first 8 (10 - 2) seconds. It won’t contain the last 2 seconds of the
stream. It seems acceptable to remove the first 2 seconds instead.
PiperOrigin-RevId: 520916464
- To support looping EditedMediaItemSequences, we need a way to tell the
AssetLoader that a sample couldn't be consumed and that it should retry
later. This is necessary in case we don't know yet whether the looping
sequence should load more samples because the other sequences haven't
made sufficient progress yet.
- The decision on whether to consume a sample is based on its timestamp
so it needs to be available.
PiperOrigin-RevId: 516546026
Split test/ TransformerEndToEndTest into SingleMediaItemEndToEndTest and
SingleSequenceEndToEndTest to reduce the file size and split the tests
by category.
PiperOrigin-RevId: 515039502
All audio tracks should either all be transcoded or all be transmuxed.
Same for video tracks.
To achieve this, simplify the behaviour of transmuxAudio/Video.
PiperOrigin-RevId: 513809287
If the Metadata passed to SegmentSpeedProvider is null, then the
SegmentSpeedProvider will always return 1f from getSpeed.
Initializing a SpeedChangingAudioProcessor requires a SpeedProvider.
Once configured,this audioProcessor is always active, so buffers are
passed through it. Because getSpeed is always 1, the processor performs
a no-op, but still has to do a buffer copy for each buffer.
By not initializing the audio processor when metadata is null, this
copy can be skipped and the audio pipeline is more performant.
Note: This change does not affect the multiple media-item case, which
is not supported with speed changes, as per Transformer API
documentation.
PiperOrigin-RevId: 513261811
- Split the transmux setting into transmuxAudio and transmuxVideo. This
is more flexible for apps and will also be useful for unit testing
(particularly as we can't test video transcoding on Robolectric at the
moment).
- Move these settings to Composition. It makes sense for these settings
to be next to forceAudioTrack. Apps may also want to set these
settings based on the current Composition's MediaItems.
- Add a Composition.Builder because Composition now contains a few
optional fields.
PiperOrigin-RevId: 511708618
- Add silent audio when the output contains an audio track but the
current MediaItem doesn't have any audio.
- Add an audio track when generateSilentAudio is set to true.
PiperOrigin-RevId: 511005887
Rename ScaleToFitTransformation to ScaleAndRotateTransformation.
This better represents the operations that can be accomplished using this
effect. The name was originally named ScaleToFit* because it's not obvious how
to scale to fit using OpenGL, and this effect handled the scaling to fit in a way that no other MatrixTransformations did.
However, it's hard to discover how to rotate when skimming names of effects, so
it's probably more useful to convey that this effect rotates, than that it
scales to fit.
PiperOrigin-RevId: 510480078
This allows us to fix usage of HDR_MODE_EXPERIMENTAL_FORCE_INTERPRET_HDR_AS_SDR.
Before, this was checked in the VideoSamplePipeline, which no longer decides on the decoder configuration input format.
PiperOrigin-RevId: 510142097
- For single-asset, the behavior stays the same. Transcode if and only
if it's necessary,
- For constrained multi-asset, always transcode, except if the setter to
transmux is set. This is to avoid failing if a MediaItem that doesn't
require transcoding is followed by a MediaItem that does require
transcoding.
PiperOrigin-RevId: 508097798
ParcelFileDescriptor is quite specific and removing it allows to have
less overloads of startTransformation.
A follow-up CL will undeprecate the overload that takes a MediaItem.
PiperOrigin-RevId: 505670321
Also add a Builder to EditedMediaItem to avoid having a constructor with
many optional parameters, or a chain of constructors.
PiperOrigin-RevId: 504588544
Transformer convention has been to avoid splitting into packages unless
necessary.
See https://jlbp.dev/JLBP-2: "Prefer fewer packages over more packages
to avoid unnecessarily publicizing internal details, since any
dependency across package boundaries needs to be public.
PiperOrigin-RevId: 503463294
