When running on emulators, ExternalTextureManager needs a longer timeout for
forcing EOS, but we didn't catch a device name running on blaze: `generic_x86`
PiperOrigin-RevId: 565513152
Organize logic a bit by splitting logic about how we draw using OpenGL onto a
texture, out from the larger class, which has lots of logic discussing how we select
frames and streams.
No functional change intended, but a few method calls are shuffled around to
simplify things.
PiperOrigin-RevId: 565426225
As discussed offline, this is important for users of the class, and not all users may choose to read method javadoc, so best to make sure it's visible by leaving it at the class-level.
PiperOrigin-RevId: 565057677
That said, only SDR is supported for now, so this will always throw if
HDR is input. This will also throw if different ColorInfo values are input
because color SDR mixing (ex. between sRGB and BT709) is not yet supported.
PiperOrigin-RevId: 563457729
Instead of invoking "undefined behavior" may occur when a class is used
incorrectly, which is clear but not very helpful to callers, specify what Apps
or users of a class should do when issues occur.
Apply this to Compositor as well, where this otherwise might seem unclear.
PiperOrigin-RevId: 563406219
Overlays may be overlaid over:
* In VideoFrameProcessor, videos or images (or texture input).
* In Compositor, other videos.
In Compositor, Overlays may consist of video, so it could be confusing
for videoFrameAnchor to contrast with overlayAnchor.
Also, rename overlayAnchor to overlayFrameAnchor, since it's modifying
the anchor in the overlay's frame, so this name seems slightly more precise.
PiperOrigin-RevId: 562004292
Implement VideoCompositor support of:
* Different input and output sizes
* CompositorSettings, to customize output size based on input texture sizes
* OverlaySettings, to place an input frame in an arbitrary position on
the output frame.
Also, refactor Overlay's matrix logic to make it more reusable between
Compositor and Overlays
PiperOrigin-RevId: 561931854
Split this logic out from `OverlayShaderProgram`, to share logic like
this with `VideoCompositor`.
Also, remove the unused `context` parameter from `OverlayShaderProgram`
`VideoCompositor` will reuse `OverlayMatrixProvider`'s storing of the matrices,
though it may generate the matrices differently, since `OverlayShaderProgram`
applies this in the fragment shader, whereas `VideoCompositor` applies this in the
vertex shader.
PiperOrigin-RevId: 560142164
The actual logic scales the alpha, instead of setting it, so rename
this to what it does.
We would also prefer alpha to be scaled here, to not lose alpha information
(for example for PNGs with transparent regions).
PiperOrigin-RevId: 560121708
The first call to method `registerInputStream` doesn't block.
Later successive calls to the method blocks until the previous register call
finishes.
PiperOrigin-RevId: 559694490
Replace Queue<Long> with LongArrayQueue which provides queue semantics
for long primitives. LongArrayQueue is forked from IntArrayQueue which
in turn was forked from Androidx CircularIntArray.
IntArrayQueue is deleted and we now use CircularIntArray directly from
Androidx Collection.
PiperOrigin-RevId: 559129744
In case BaseShaderProgram implementations have an expensive
configure() call, we can only call configure() when absolutely necessary,
aka when the input size changes.
To reduce scope, this doesn't reduce the amount of
configure() calls that may be made outside this class, in
DefaultFrameDroppingShaderProgram and
FinalShaderProgramWrapper.
PiperOrigin-RevId: 558754314
In addition to this being how Alpha should be handled, tests do fail under
API 29 without this CL. Unfortunately while these tests do fail under API 29
without this CL, we currently only run this test on API 33 emulators, so we
didn't catch this failure earlier (until compositor tests on mh failed on all
devices under API 29 and succeeded on all at or over 29).
PiperOrigin-RevId: 557781757
`BlankFrameProducer` was originally internal to
`DefaultVideoFrameProcessorVideoFrameRenderingTest`. We later copied it in
c221958889. This CL converges the usages.
PiperOrigin-RevId: 555953620
This effect can scale Alpha values.
This is needed for testing Compositor, and coincidentally was requested
by a partner in the past. This also will generally be more useful in
full multi-asset, where we may want to have one input have different alpha
values than another
Also, update fragment_shader_copy_es2.glsl to not throw away alpha values.
This means ThumbnailStripShaderProgram (the only place this glsl is used) will
consider input alpha and output alpha appropriately
PiperOrigin-RevId: 555915092
This adds an effect to change the speed of the video track. The speed
of the audio track can already be changed using a SonicAudioProcessor.
Issue: androidx/media#559
PiperOrigin-RevId: 555155418
When implementing pause and resume, overlay appears as black frame on resume, suggesting the texture isn't being remade properly due to lack of proper deletion.
PiperOrigin-RevId: 554832580
The test file decompresses to about 48 MB, which can cause an
`OutOfMemoryError` on lower end devices.
Also, a test appears to apply a rotation to this bitmap without downscaling
first, leading to exceeding a maximum texture width/height of 4096 on some
devices.
Replace the test file with one that has a quarter of the original's
width/height so that these tests can pass reliably even on lower end
devices.
PiperOrigin-RevId: 554507185
This means we now require 2+ input frames per input, and compare the primary
stream timestamp with secondary stream timestamps in order to select the
correct output timestamp. We also must release frames and back-pressure as
soon as possible to avoid blocking upstream VFPs.
Also, improve signalling of VFP onReadyToAcceptInputFrame
PiperOrigin-RevId: 553448965
This allows for custom implementations of this interface, like
a TestVideoCompositor or partner-implemented implementation
PiperOrigin-RevId: 552541631
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).
This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)
- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
- We bypass the input manager
- The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
- The generated frame is sent to texture output and in turn saved to bitmaps
- We then compare the generated bitmap with the goldens
PiperOrigin-RevId: 551795770
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.
PiperOrigin-RevId: 550839156
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.
onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors
PiperOrigin-RevId: 549969933
The VideoFrameProcessingTaskExecutor already handles wrapping
GlUtil.GlException errors into VideoFrameProcessingExceptions.
Therefore, stop wrapping this ourselves, so that errors are
attributed to the correct line number with less indirection.
PiperOrigin-RevId: 549959258
Alters the OverlayShaderProgram implementation to support rotations, however we need to apply the transformations separately in order for them to work as expected so the matrix is removed from the interface in favour of explicit methods.
Adds a rotation test to ensure this ability doesn't regress
PiperOrigin-RevId: 549890847
For now, just identical timestamps. A follow-up CL will allow for different
framerates. Prior tests already send identical timestamp frames, so no new
tests were added
PiperOrigin-RevId: 548676759
Before this CL, the text with a scale span would wrap text, with because the scale wasn't taken into account when measuring the width of the overlay.
PiperOrigin-RevId: 548123626
* Allow more than one input bitmap at a time.
* Allow Compositor to take in and set up an Executor. Otherwise,
Compositor resources may be created on one thread and accessed on another.
* Add a Compositor TestRunner to reuse test code more.
* Update VideoFrameProcessingTaskExecutor to use a new onError listener, so
that it's more reusable in non-DVFP contexts, like for Compositor.
PiperOrigin-RevId: 547206053
Frame buffers are focused, not current. EGL Contexts may be current but this
is a prerequisite for focusing a framebuffer, and this alone isn't
sufficient to make a bitmap.
Clear this up as it can make some complex tests harder to understand, if the
methods needed don't match up in naming.
PiperOrigin-RevId: 546305118
BaseGlShaderProgram and SingleFrameGlShaderProgram are technically color space agnostic. useHdr is passed to the TexturePool, which only uses the value to choose between high/regular precision. useHdr is therefore a bit misleading and should be generalized to "useHighPrecisionColorComponents."
PiperOrigin-RevId: 545726629
This allows us to simplify and reduce thread initialization, hops and the
synchronization cost associated with them, especially when more and more VFPs
are used.
PiperOrigin-RevId: 545622169
Providing the sync token in the api allows the client to decide which waiting method they would like to use depending on the use case, allowing them to optimise if possible.
PiperOrigin-RevId: 543997311
MP4 edit lists sometimes ask to start playback between two samples.
If this happens, we currently change the timestamp of the first
sample to zero to trim it (e.g. to display the first frame for a
slightly shorter period of time). However, we can't do this to audio
samples are they have an inherent duration and trimming them this
way is not possible.
#minor-release
PiperOrigin-RevId: 543420218
By passing this class where it's needed, implementations don't need to store it
in a field (reducing boilerplate) and it's clearer that it can't be unset when
needed.
PiperOrigin-RevId: 542823522
To more accurately describe what they do, especially as Compositor will
starts to use more contexts or threads, and it's important to know what
needs to be reset/recreated/focused before what methods.
PiperOrigin-RevId: 541010135
Previously, TextureMangers have a method to signal ending of a current input
stream, and a method to end the **entire input**. The responsibility of both
methods are not easy to document, understand and read.
With the new design,
- Only `TextureManager.signalEndOfCurrentInputStream()` is kept
- It's called for every MediaItem in the sequence, include the final one
- FinalWrapper now takes explicit signal that frame processing is ending,
rather than relying on the return value of `onCurrentInputStreamProcessed()`
- On DVFP receiving EOS from the pipeline, it signals FinalWrapper the stream
is ending, **before** signaling the input switcher, so that FinalWrapper is
able to end the stream when the onCurrentInputStreamEnded signal eventually
reaches FinalWrapper
PiperOrigin-RevId: 540856680
This helps debug bitmap tests. Without this CL, I typically
need to go into the file to find the test name and absolute
path. With this CL, I can just adb pull the printed file path,
which is much simpler.
PiperOrigin-RevId: 540539042
The existing NullableType has been deprecated 5 years ago and causes
crashes in Kotlin apps because Kotlin doesn't recognize this annotation
as a nullable type annotation.
While we can't align on a single @Nullable annotation yet, we can at
least replace this one by JSR305's @Nonnull(MAYBE) as it fulfils all
requirements, including full Kotlin compatiblity. To avoid the
cumbersome name, we can redefine it as our own @NullableType
annotation. (We can't use @Nullable to avoid name clashes with the main
@Nullable annotation from AndroidX)
Issue: google/ExoPlayer#6792
PiperOrigin-RevId: 540497469
Partially roll forward of 5c29abbbf4, and adds some extra logic
Changes to the original CL
The original logic (a66f08ba97) fails in the following case:
> This is only seem on emulators.
- EOS is sent to ExtTexMgr
- The timer starts
- One frame arrives on SurfaceTexture, reset the timer
- The frame is sent for processing, now `availablFrames == 0`
- One frame arrives on Surface, reset the timer
- The frame is kept on SurfaceTexture for the downstream shader doesn't have
capacity, `availablFrames == 1`
- Timer times out as the downstream processor doesn't report being able to take
another frame.
- Although there's a frame available on the SurfaceTexture
This is solved by having the force EOS logic clear all the frames that the
SurfaceTexture holds. This also ensures the first frame dequeued from the next
stream isn't from the previous stream.
PiperOrigin-RevId: 540023359
This methods currently registers and queues the bitmaps Will need to have a method that just queues bitmaps in future CLs.
#minor-release
PiperOrigin-RevId: 539605628
*** Original commit ***
Add a timer to end a video stream prematurely in ExtTexMgr
***
This has been submitting for more than 1.5hrs. "This presubmit is running slowly because you have been throttled by Build Queue due to using too much of your Product Area's quota."
adding NO_SQ as this is a pure rollback
PiperOrigin-RevId: 539135970
To ensure no regressions for the potentially confusing pipeline of:
* HDR electrical -> SDR linear EOTF+OOTF, and
* SDR linear -> SDR electrical OETF
PiperOrigin-RevId: 538741079
`outputSurfaceInfoChanged` is not reset when `defaultShaderProgram` is null.
That is, on the first time `ensureConfigured()` is called with output size
changed, `outputSurfaceInfoChanged` is not set to false after creating the
`defaultShaderProgram`, and `defaultShaderProgram` will be created again on the
second time `ensureConfigured()` is called.
PiperOrigin-RevId: 537870404
This future.get() duplicates the wait done in
singleThreadExecutorService.awaitTermination(). If awaitTermination times out, this future.get() would also result in unnecessary blocking.
PiperOrigin-RevId: 536442153
Also, document that texture output disables manual frame release.
In the past, texture output would lead to surface output methods throwing. Now,
they're simply no-ops instead.
PiperOrigin-RevId: 534894168
Docs:
* https://developer.android.com/build/publish-library/configure-pub-variants#single-pub-var
* https://developer.android.com/reference/tools/gradle-api/8.0/com/android/build/api/dsl/PublishingOptions
This resolves the following warning from Android Gradle Plugin 7.2.2 and
helps unblock the upgrade to AGP 8.0.1:
> Software Components will not be created automatically for Maven
> publishing from Android Gradle Plugin 8.0. To opt-in to the future
> behavior, set the Gradle property
> `android.disableAutomaticComponentCreation=true` in the
> `gradle.properties` file or use the new publishing DSL.
>
> Affected Modules: `lib-cast`, `lib-common`, `lib-container`,
> `lib-database`, `lib-datasource`, `lib-datasource-cronet`,
> `lib-datasource-okhttp`, `lib-datasource-rtmp`, `lib-decoder`,
> `lib-effect`, `lib-exoplayer`, `lib-exoplayer-all (legacy)`,
> `lib-exoplayer-dash`, `lib-exoplayer-hls`, `lib-exoplayer-ima`,
> `lib-exoplayer-rtsp`, `lib-exoplayer-smoothstreaming`,
> `lib-exoplayer-workmanager`, `lib-extractor`, `lib-media2 (legacy)`,
> `lib-mediasession (legacy)`, `lib-muxer`, `lib-session`,
> `lib-transformer`, `lib-ui`, `lib-ui-leanback`, `test-utils`,
> `test-utils-robolectric`
Issue: androidx/media#409
PiperOrigin-RevId: 533464133
This is a pre-requisite for the Android Studio upgrade assistant to
upgrade from AGP 7.2.2 to 8.0.1, otherwise it fails and complains
this is missing.
Issue: androidx/media#409
PiperOrigin-RevId: 533463246
This is blocking the upgrade to AGP 8.0.1. The Android Studio Upgrade
Assistant refuses to run with the error:
> Modules have the same package for their `main` and `androidTest`
> artifacts
>
> The package specifications in `AndroidManifest.xml` files define the
> same package for the `main` and `androidTest` artifacts, in the
> following modules: `effect`, `muxer`, `transformer`,
`test_exoplayer_abr`, `test_exoplayer_performance`, and `container`.
This change also adds the `.test` suffix to `test/AndroidManifest.xml`
files where it's missing, because although it's not mentioned in the
Upgrade Assistant error it does cause problems later.
```
> Incorrect package="androidx.media3.database" found in source AndroidManifest.xml: libraries/database/src/test/AndroidManifest.xml.
Setting the namespace via the package attribute in the source AndroidManifest.xml is no longer supported.
Recommendation: remove package="androidx.media3.database" from the source AndroidManifest.xml: libraries/database/src/test/AndroidManifest.xml.
```
Issue: androidx/media#409
PiperOrigin-RevId: 533460180
This allows us to avoid needing a reference to the VideoFrameProcessor, which
can be especially difficult if an App only has a reference to the
VideoFrameProcessor.Factory it passes into Transformer/ExoPlayer.
PiperOrigin-RevId: 533205983
Have the FinalShaderProgramWrapper / VideoFrameProcessor texture
output access textures provided through a texture pool, that
recycles used textures.
Also, add the TexturePool interface to generally re-use textures.
PiperOrigin-RevId: 532754377
Propagate the "end of current stream" signal directly after queueing the
last frame, instead of waiting for the next onReadyToAcceptInputFrame()
call.
PiperOrigin-RevId: 532739462
Allow the VideoFrameProcessor to output multiple textures at a time, so that
lifetime of textures is up to the consumer calling VFP.releaseOutputFrame.
The FinalShaderProgramWrapper also has a new maxCapacity limit added, to ensure
the a reasonable amount of textures is used and avoid using up memory.
PiperOrigin-RevId: 532094256
When exporting compositions with multiple images in a row, transformation could
get stuck if a shader was ready to accept input when end-of-stream was already
signaled and queued from upstream. Fix accounting for the downstream capacity.
Manually tested on concatenations with several images and several videos in a
row, by adding logging and verifying the capacity updates as expected across
edited media item transitions.
PiperOrigin-RevId: 532088793
The output end-of-stream notification from the last shader could theoretically
arrive before the latch for detecting it is created, which might cause waiting
on the latch indefinitely. Create the latch before signaling end of stream so
that it's guaranteed to be set before the end-of-stream signal arrives.
PiperOrigin-RevId: 532056472
For DefaultVideoFrameProcessorVideoFrameRenderingTest: `registerInputStream` sets the textureManager, so reordering makes sure the textureManager is set before you set the frame info. This is important for texture input, where the frame info provides the width and height.
PiperOrigin-RevId: 529753404
This means that comments like `//copybara:media3-only` are now detected
(no space between `//` and `copybara`) which will ensure that lines like
this are correctly transformed for the media3 and exoplayer2 GitHub
repos:
aa4e008014/library/effect/build.gradle (L33)
PiperOrigin-RevId: 527919649
Also make FinalShaderProgramWrapper always receive internal texture.
This means it does not sample from a input texture, and its input color is
always linear, hence the input type does not matter.
PiperOrigin-RevId: 527869045
*** Original commit ***
Effect: glFlush instead of glFinish on tex output
This is much faster (~2-3x) than glFlush. While there's a risk that GL commands
queued to the GL server may not be complete by the time non-GL commands access
the texture, this should be unlikely as we only access the texture from GL.
If we see stability issues in the future, we can reconsider and move this back
to glFinish (or GL synchronization mechanisms like fences, which are more
complex)
***
PiperOrigin-RevId: 527848094
renderOutputFrame actually renders frames to an output surface. We'll soon have
a releaseOutputFrame method, that would release resources associated with an
output time, so rename this to disambiguate the two methods.
Also rename onOutputFrameAvailable to onOutputFrameAvailableForRendering, to
make it clear this is not available for "release"
This change should be a renaming-only change and have no functional differences.
PiperOrigin-RevId: 527844947
This is much faster (~2-3x) than glFlush. While there's a risk that GL commands
queued to the GL server may not be complete by the time non-GL commands access
the texture, this should be unlikely as we only access the texture from GL.
PiperOrigin-RevId: 527641520
Add `VideoFrameProcessor.registerInputStream()` to signal a new type of input.
And `InputHandler.signalEndOfCurrentInputStream()` to signal to `InputHandler`
partial input stream completion.
Fully processed means after FinalShaderProgramWrapper releases the last frame.
PiperOrigin-RevId: 527356646
Allow the VideoFrameProcessor to output to a texture without an output surface.
Tested by updating texture output tests to no longer output to a surface.
PiperOrigin-RevId: 527244605
