Covers all permutations&combinations of:
* AV asset.
* AV asset with effects.
* AV asset with silence.
* AV asset with silence and effects.
Note that the video is not relevant (therefore transmuxed), but is
needed for silence generation.
PiperOrigin-RevId: 558734593
We will need to build upon original composition
for pause and resume feature and will have to set modified sequences.
In the next CL, will add method to set sequences on builder and will
probably remove it from the constructor parameter (to avoid providing
two APIs for same thing).
PiperOrigin-RevId: 558127334
I wasn't able to reproduce the error in the bug by running videoTranscoding_withTextureInput_completesWithCorrectFrameCountAndDuration
100 times, but I figured this added debug info may help the next time the
issue is seen.
PiperOrigin-RevId: 558118443
In addition to this being how Alpha should be handled, tests do fail under
API 29 without this CL. Unfortunately while these tests do fail under API 29
without this CL, we currently only run this test on API 33 emulators, so we
didn't catch this failure earlier (until compositor tests on mh failed on all
devices under API 29 and succeeded on all at or over 29).
PiperOrigin-RevId: 557781757
Rename AudioMixerImpl to DefaultAudioMixer.
Removes the AudioMixerImpl specific getOutputAudioFormat, as the caller
defines and sets this.
PiperOrigin-RevId: 555887722
To ensure that for each output bitmap from the compostor, the right input
timestamps were used.
Only applied on a subset of tests to avoid needing to upload+maintain
too many files/size in the test binary, especially when it would test
duplicate behavior
PiperOrigin-RevId: 555222530
This means we now require 2+ input frames per input, and compare the primary
stream timestamp with secondary stream timestamps in order to select the
correct output timestamp. We also must release frames and back-pressure as
soon as possible to avoid blocking upstream VFPs.
Also, improve signalling of VFP onReadyToAcceptInputFrame
PiperOrigin-RevId: 553448965
ExoPlayer queues the EOS buffer to the decoder with offset/size/timestamp all
equal to zero, and a EOS flag.
69769c77b3 set TIME_END_OF_SOURCE on the EOS buffer from the extractor.
Queueing the EOS buffer to the decoder with TIME_END_OF_SOURCE causes some
decoders to output wrong timestamps in its output.
This CL replicates what ExoPlayer does in Transformer.
PiperOrigin-RevId: 553104213
This allows for custom implementations of this interface, like
a TestVideoCompositor or partner-implemented implementation
PiperOrigin-RevId: 552541631
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).
This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)
- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
- We bypass the input manager
- The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
- The generated frame is sent to texture output and in turn saved to bitmaps
- We then compare the generated bitmap with the goldens
PiperOrigin-RevId: 551795770
Move shared test logic to the test runner.
This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.
PiperOrigin-RevId: 551536438
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.
PiperOrigin-RevId: 550894935
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.
Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.
Regression tests reproduce failure without fix, but pass with it.
PiperOrigin-RevId: 550853714
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.
PiperOrigin-RevId: 550839156
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.
onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors
PiperOrigin-RevId: 549969933
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.
Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.
PiperOrigin-RevId: 548994584
This serves as a sort of test with more frames for the compositor for now
(before more varied video system tests come later when integrating with
Transformer), showing it doesn't error out and outputs the right
amount of frames.
Due to the VFPTestRunner having a 5s timeout, and mostly due to presubmit
emulators being very slow with OpenGL, there is a sort of limitation
to how many frames this type of test can have, depending on the test target
PiperOrigin-RevId: 548159762
SingleInputVideoGraph implements GraphInput now, so the asset loaders would
interface directly with SIVG, rather than VideoSampleExporter. This is to pave
way for multi-asset video processing.
PiperOrigin-RevId: 547561042
* Allow more than one input bitmap at a time.
* Allow Compositor to take in and set up an Executor. Otherwise,
Compositor resources may be created on one thread and accessed on another.
* Add a Compositor TestRunner to reuse test code more.
* Update VideoFrameProcessingTaskExecutor to use a new onError listener, so
that it's more reusable in non-DVFP contexts, like for Compositor.
PiperOrigin-RevId: 547206053
durationUs is almost always going to be a larger number than the sample
rate, so pass as the main value, rather than the multiplier.
PiperOrigin-RevId: 547193927
TransformerAndroidTestRunner#run will throw any exception that occurs,
so there is no need to assert the result exception is null.
PiperOrigin-RevId: 546923823
Frame buffers are focused, not current. EGL Contexts may be current but this
is a prerequisite for focusing a framebuffer, and this alone isn't
sufficient to make a bitmap.
Clear this up as it can make some complex tests harder to understand, if the
methods needed don't match up in naming.
PiperOrigin-RevId: 546305118
Add documentation for threading requirements at the class level (in
addition to existing documentation on the methods) to improve
discoverablility. Also fix a couple of nits in the javadoc (US English
spelling, avoid passive voice) and in `OnInputFrameProcessedListener`.
PiperOrigin-RevId: 546303732
This allows us to simplify and reduce thread initialization, hops and the
synchronization cost associated with them, especially when more and more VFPs
are used.
PiperOrigin-RevId: 545622169
On a MediaItem change, the input Format (and Effects to apply) may be
different. Therefore the AudioProcessingPipeline must be reconfigured
to determine what processing is active, and what the AudioFormat of the
data output is. In the event that it is different, additional
AudioProcessor instances must be used to ensure the encoder will still
be able to accept the audio buffers.
PiperOrigin-RevId: 544338451
Providing the sync token in the api allows the client to decide which waiting method they would like to use depending on the use case, allowing them to optimise if possible.
PiperOrigin-RevId: 543997311
This fixes a broken javadoc link and makes it possible to customize media
source creation via the asset loader factory without also having to
implementing a decoder factory from scratch.
PiperOrigin-RevId: 543995538
Goal of tests (SequenceExportTest) that use this media is for the
silence and the media to match exactly with audio format, however
`sample_with_increasing_timestamps.mp4` had a different sample rate.
testvid_1022ms.mp4: channel count = 2, sample rate = 44100.
PiperOrigin-RevId: 543458948
MP4 edit lists sometimes ask to start playback between two samples.
If this happens, we currently change the timestamp of the first
sample to zero to trim it (e.g. to display the first frame for a
slightly shorter period of time). However, we can't do this to audio
samples are they have an inherent duration and trimming them this
way is not possible.
#minor-release
PiperOrigin-RevId: 543420218
FallbackListener.onTransformationRequestFinalized() is called from the
AssetLoader thread for audio, and from the GL thread for video.
PiperOrigin-RevId: 542851284
With the upcoming "handle format changes" CL, stereo -> mono audio
would add an AudioProcessor. Robolectric decodes output encoded data,
which crashes some AudioProcessors because the number of frames may not
be an integer.
PiperOrigin-RevId: 542568875
The effects pipeline must receive images in the sRGB colorspace due to the color transfers applied in the shaders. Currently the burden to making sure images are in the right colorspaces falls onto apps. This CL ensures that this is not the case anymore.
PiperOrigin-RevId: 542323613
Audio only tests are now using RAW audio where possible, which is
passed through the Robolectric decoders/encoders, and can be handled by
the AudioProcessor instances accurately.
PiperOrigin-RevId: 541648853
To more accurately describe what they do, especially as Compositor will
starts to use more contexts or threads, and it's important to know what
needs to be reset/recreated/focused before what methods.
PiperOrigin-RevId: 541010135
Issue: When running the Transformer related test cases, the tests are flaky
because the order in which audio and video samples are interleaved seems to
differ in few instances.
Root cause: When running a transformation the sample producer (Asset loader)
and sample consumer (Sample pipeline) both runs on different thread and
theoretically there is no reason for behaviour to be deterministic because
the number of samples produced/written depends on how fast individual thread
works. So it is indeed surprising that test somehow worked deterministically in
majority of instances (may be something to do with Robolectric environment).
Solution: Since we don't expect the order of sample interleaving to be deterministic, make the dumping logic deterministic where all the video
samples will be collected and then dumped together (similarly for audio). This would mean we won't be able to see the interleaving so for that we need to
add separate test case verifying the interleaving logic only.
Pending: Test case for interleaving.
PiperOrigin-RevId: 540930871
The existing NullableType has been deprecated 5 years ago and causes
crashes in Kotlin apps because Kotlin doesn't recognize this annotation
as a nullable type annotation.
While we can't align on a single @Nullable annotation yet, we can at
least replace this one by JSR305's @Nonnull(MAYBE) as it fulfils all
requirements, including full Kotlin compatiblity. To avoid the
cumbersome name, we can redefine it as our own @NullableType
annotation. (We can't use @Nullable to avoid name clashes with the main
@Nullable annotation from AndroidX)
Issue: google/ExoPlayer#6792
PiperOrigin-RevId: 540497469
The test transcodes four media items with distinct effects, keeping one frame
each; extracts the four frames in the produced video, and compares them against
the expected results.
PiperOrigin-RevId: 539697344
This methods currently registers and queues the bitmaps Will need to have a method that just queues bitmaps in future CLs.
#minor-release
PiperOrigin-RevId: 539605628
To ensure no regressions for the potentially confusing pipeline of:
* HDR electrical -> SDR linear EOTF+OOTF, and
* SDR linear -> SDR electrical OETF
PiperOrigin-RevId: 538741079
208eefc0fd introduced using `DefaultDecoderFactory.getDecoderInfo(format) != null` caused certain tests not to be skipped when they were expected to be, creating more mh failures.
PiperOrigin-RevId: 537820370
ExoPlayer extractors (backing `MetadataRetriever`) now parse the color format
from the bitstream so using `MetadataRetriever` should be an equivalent but
more lightweight way to verify the color info.
Also remove try/catch blocks in test code calling into these methods, and add
skipping based on decoder capabilities in the cases where it was missing.
PiperOrigin-RevId: 537789483
In some cases the codec selected for decoding has a different MIME type than
the media. In thoses cases Transformer continued to use the media's MIME type
and that caused codec configuration failures.
Removed `EncoderUtil.findCodecForFormat()` as we stopped using the method it
uses for finding a codec. Plus, the method is only used in the test.
See also `MediaCodecUtil.getALternativeCodecMimeType()`.
PiperOrigin-RevId: 536683663
Tentative/experimental value to reduce codec timeouts. We will reconsider using a larger limit after seeing whether this really does reduce error rate.
PiperOrigin-RevId: 534491615
Mp4Muxer already supports writing Mp4LocationData so added that
as supported Metadata entry.
Support for more Metadata entries will be added in upcoming CLs.
PiperOrigin-RevId: 534473866
Earlier metadata was written multiple times as it came.
With new changes, all the distinct metadata entries will
get collected and will be written at once in the end.
PiperOrigin-RevId: 534088401
Docs:
* https://developer.android.com/build/publish-library/configure-pub-variants#single-pub-var
* https://developer.android.com/reference/tools/gradle-api/8.0/com/android/build/api/dsl/PublishingOptions
This resolves the following warning from Android Gradle Plugin 7.2.2 and
helps unblock the upgrade to AGP 8.0.1:
> Software Components will not be created automatically for Maven
> publishing from Android Gradle Plugin 8.0. To opt-in to the future
> behavior, set the Gradle property
> `android.disableAutomaticComponentCreation=true` in the
> `gradle.properties` file or use the new publishing DSL.
>
> Affected Modules: `lib-cast`, `lib-common`, `lib-container`,
> `lib-database`, `lib-datasource`, `lib-datasource-cronet`,
> `lib-datasource-okhttp`, `lib-datasource-rtmp`, `lib-decoder`,
> `lib-effect`, `lib-exoplayer`, `lib-exoplayer-all (legacy)`,
> `lib-exoplayer-dash`, `lib-exoplayer-hls`, `lib-exoplayer-ima`,
> `lib-exoplayer-rtsp`, `lib-exoplayer-smoothstreaming`,
> `lib-exoplayer-workmanager`, `lib-extractor`, `lib-media2 (legacy)`,
> `lib-mediasession (legacy)`, `lib-muxer`, `lib-session`,
> `lib-transformer`, `lib-ui`, `lib-ui-leanback`, `test-utils`,
> `test-utils-robolectric`
Issue: androidx/media#409
PiperOrigin-RevId: 533464133
This is a pre-requisite for the Android Studio upgrade assistant to
upgrade from AGP 7.2.2 to 8.0.1, otherwise it fails and complains
this is missing.
Issue: androidx/media#409
PiperOrigin-RevId: 533463246
This is blocking the upgrade to AGP 8.0.1. The Android Studio Upgrade
Assistant refuses to run with the error:
> Modules have the same package for their `main` and `androidTest`
> artifacts
>
> The package specifications in `AndroidManifest.xml` files define the
> same package for the `main` and `androidTest` artifacts, in the
> following modules: `effect`, `muxer`, `transformer`,
`test_exoplayer_abr`, `test_exoplayer_performance`, and `container`.
This change also adds the `.test` suffix to `test/AndroidManifest.xml`
files where it's missing, because although it's not mentioned in the
Upgrade Assistant error it does cause problems later.
```
> Incorrect package="androidx.media3.database" found in source AndroidManifest.xml: libraries/database/src/test/AndroidManifest.xml.
Setting the namespace via the package attribute in the source AndroidManifest.xml is no longer supported.
Recommendation: remove package="androidx.media3.database" from the source AndroidManifest.xml: libraries/database/src/test/AndroidManifest.xml.
```
Issue: androidx/media#409
PiperOrigin-RevId: 533460180
This allows us to avoid needing a reference to the VideoFrameProcessor, which
can be especially difficult if an App only has a reference to the
VideoFrameProcessor.Factory it passes into Transformer/ExoPlayer.
PiperOrigin-RevId: 533205983
This device failed on HdrEditingTest's exportAndTranscode_hlg10File_whenHdrEditingUnsupported_toneMapsOrThrows
before this CL, and succeeds on that test after this CL.
PiperOrigin-RevId: 532796897
This is a failure only in SSIM, so it seems unlikely we'll prioritize this over
other work or bugs soon. Suppress test failures to reduce triage burden.
PiperOrigin-RevId: 532200729
Allow the VideoFrameProcessor to output multiple textures at a time, so that
lifetime of textures is up to the consumer calling VFP.releaseOutputFrame.
The FinalShaderProgramWrapper also has a new maxCapacity limit added, to ensure
the a reasonable amount of textures is used and avoid using up memory.
PiperOrigin-RevId: 532094256
In the past, the SequenceAssetLoader was released in TransformerInternal
when the export ended.
fc539da061 was made to release the SequenceAssetLoader earlier, when
loading ended. This was causing player release timeouts because the last
AssetLoader in the sequence was released before the SamplePipelines (see
0b40bc37ab for more information).
The code that was releasing the SequenceAssetLoader was first commented
out because we didn't have an understanding of what was happening.
This change removes the early SequenceAssetLoader release all together.
It doesn't have any effect as this code was already commented out.
PiperOrigin-RevId: 532065673
*** Original commit ***
Rollback of 65d5132f76
*** Original commit ***
Create InAppMuxer in transformer
To use the InAppMuxer, the client needs to pass InAppMuxer Factory.
***
***
PiperOrigin-RevId: 531470081
The texture input tests in TransformerEndToEndTest were not passing on
Pixel 7. Implemented a fix and fixed other minor threading issues I
spotted while looking at the code.
PiperOrigin-RevId: 531141659
*** Original commit ***
Create InAppMuxer in transformer
To use the InAppMuxer, the client needs to pass InAppMuxer Factory.
***
PiperOrigin-RevId: 531056436
Tone-mapping an HDR video with MediaCodec on sm-s908u1 was timing out.
The reason for that is that the decoder was dropping frames, and the
ExternalTextureManager was therefore never propagating the end-of-stream
signal.
There was already a workaround for a similar issue but restricted to
sm-f936b. Removed the model check as the bug is probably present on more
devices.
PiperOrigin-RevId: 530639437
Exceptions thrown when switching AssetLoader were not reported through
the error listener. Therefore, the resources were not released and the
export was not ending.
PiperOrigin-RevId: 530626300
If tone-mapping was requested for a device supporting HDR encoding,
isToneMapped was false in VideoSamplePipeline.EncoderWrapper. This was
causing the encoder to expect HDR.
Also did some renamings to improve readability
PiperOrigin-RevId: 530584010
similar error to 9baa6f6be5L `registerInputStream` sets the textureManager, so reordering makes sure the textureManager is set before you set the frame info. This is important for texture input, where the frame info provides the width and height.
PiperOrigin-RevId: 530579584
When we created androidTests, in the past, they always had a URI pointing to a resource, therefore we always had a URI scheme. With texture input, this will not longer be the case (EditedMediaItems's may have URI.EMPTY, which have a null scheme) so we need to check for this so tests don't falsely fail.
PiperOrigin-RevId: 528848411
renderOutputFrame actually renders frames to an output surface. We'll soon have
a releaseOutputFrame method, that would release resources associated with an
output time, so rename this to disambiguate the two methods.
Also rename onOutputFrameAvailable to onOutputFrameAvailableForRendering, to
make it clear this is not available for "release"
This change should be a renaming-only change and have no functional differences.
PiperOrigin-RevId: 527844947
Add `VideoFrameProcessor.registerInputStream()` to signal a new type of input.
And `InputHandler.signalEndOfCurrentInputStream()` to signal to `InputHandler`
partial input stream completion.
Fully processed means after FinalShaderProgramWrapper releases the last frame.
PiperOrigin-RevId: 527356646
In ASwB, all Transformer tests can be run by right-clicking on the project and
clicking "Run all tests". We cannot, however, select only some specific test
files within a project (ex. all non-analysis tests) to run tests on.
Add @Ignore to analysis tests, which are not intended to be run anyways
when determining whether Transformer is working on some device. These
tests also don't have proper skipping logic when a device doesn't support a
format, so they can't be run effectively on low-end devices anyways.
This eases manual testing, for example when debugging whether tests all pass
on a device.
When analysis tests are desired to be run, it should be easy to comment out
the @Ignore.
PiperOrigin-RevId: 527289600
Allow the VideoFrameProcessor to output to a texture without an output surface.
Tested by updating texture output tests to no longer output to a surface.
PiperOrigin-RevId: 527244605
The only dependencies that are not updated are the ones that need
to be kept in sync with other system (like Android source tree) or
would require a Kotlin dependency in common or exoplayer modules.
As a side effect, some demo apps now need a Kotlin config and some
additional modules require desugaring/multidex logic. To simplify
the setup, the desugaring and multidex steps are added to the common
config.
PiperOrigin-RevId: 527243950
This was only used in EncoderWrapper, so move createSupportedTransformationRequest
from VideoSamplePipeline to EncoderWrapper to reduce its scope.
PiperOrigin-RevId: 527226578
The production code changes are in transformer, but the tests in effect have also been updated to confirm the is no color regression `inputColorInfo.colorTransfer=C.COLOR_TRANSFER_SRGB`
PiperOrigin-RevId: 526950435
Previously, I assumed that SDR contents must be BT709, and HDR contents must be
BT2020. Turns out BT2020 is just wide-gamut, and SDR contents / transfers may be
represented in BT2020 color spaces.
Relax the check, so that we don't throw when valid BT2020 SMPTE 170M contents
are input into effects.
PiperOrigin-RevId: 526668347
