Removes glObjectsProvider from `VideoFrameProcessor` and `Effects`. Apps will set the glObjectsProvider on the DefaultVideoFrameProcessor.Factory.Builder when providing a custom DefaultVideoFrameProcessor.Factory, rather than in `Effects`.
PiperOrigin-RevId: 525169059
A new texture manager will be created for input by texture ID so this texture manager won't be the only one to handle internal (i.e. non-external) textures.
PiperOrigin-RevId: 525116460
In `ExternalTextureManager` in seemingly rare cases end of stream is signaled
at the point where a frame is currently pending processing. In that case the
video end of stream signal was lost. If the muxer timeout was enabled this
case would result in throwing an exception, but otherwise the operation would
get stuck
Add code to signal end of stream in `onInputFrameProcessed` as well, so that we
signal end of stream when the pending frame is handled.
Tested by running
`TransformerEndToEndTest.loopingTranscodedVideo_producesExpectedResult` several
times.
PiperOrigin-RevId: 524361069
In addition to the changes in 3a5c4277a7
This change essentially reverts 30e5bc9837 (Merged Jul 2022).
From this CL on, `VideoFrameProcessor` takes in non-offset, monotonically
increasing timestamps. For example, with one 5s and one 10s video,
- `VideoFrameProcessor`'s input should start from 0
- On switching to the second video (10s), the timestamp of the first frame in
the second video should be at 5s.
In ExoPlayer however, `streamOffset` is managed differently and thus needs
correction before sending the frames to `VideoFrameProcessor`:
- The timestamp of the first video is offset by a large int, so the first frame
of the first media item has timestamp (assuming) 10000000000000000
- The last frame of the first media item has 10000005000000000
- At this point the stream off set is updated to 10000005000000000
- The pts of the first frame of the second video starts from 0 again.
PiperOrigin-RevId: 523444236
Before this CL, SurfaceTexture.onFrameAvailable was used to tell whether a frame
was available in the VideoFrameProcessor's output texture. This was incorrect, as
it would rely on having the texture be written to before the
SurfaceTexture.onFrameAvailableListener is invoked, leading to null-pointer-
exceptions on timeouts.
Instead of using DefaultVideoFrameProcessor different interfaces to set that we
want to output to a texture, and get that output texture, use one interface that
sets a listener, and renders to a texture iff that listener is set. As this
listener is executed on the GL thread, this also allows us to no longer need to
expand visibility for the GL task executor and tasks.
PiperOrigin-RevId: 522362101
NPE in toneMap_hlgFrame_matchesGoldenFile and toneMap_pqFrame_matchesGoldenFile was created because a uEnableColorTransfer uniform was being created on the HDR path, when HDR shader files don't have this uniform. (they don't support disable color transfers right now)
Fix: only create the uniform when input is SDR.
manually tested on failing tests
PiperOrigin-RevId: 522002603
By having a single base class for GL shader programs we simplify the customization
of new shader programs. The concrete cases include
- Allow frame dropping in shader program
- Creating a FrameCache that selectively (based on timestamp) replays and clears
the cached content
PiperOrigin-RevId: 520322060
Previously, we always used ImageReader to read from the output of DefaultVideoFrameProcessor, for pixel tests. This has a limitation of not being
able to read HDR contents, so that we couldn't support HDR pixel tests.
Reading from a texture allows us to use glReadPixels to read from
DefaultVideoFrameProcessor, and build upon this to implement HDR pixel tests. We do
still want tests for surface output though, because real use-cases only will output
to Surfaces.
Also, add some tests for outputting to textures, since this test infrastructure is
a bit complex.
PiperOrigin-RevId: 519786535
Since the composite matrix is ultimately rewritten to, we need to ensure it's cleared (to identity) before update.
Test plan: use an effect as a time based approach and see that the effect no longer clips
PiperOrigin-RevId: 518886623
Reduce nesting by using an early return. Also, rename
`outputSizeOrRotationChanged` to `outputChanged`, because this also applies to when the output surface changes.
Also, update local variable initialization, add some javadoc, and remove unneeded
local variable
PiperOrigin-RevId: 518241708
d288891c77 added clearing depth buffers to GLUtil, so there is no need to have allow apps to have a custom clearOutputFrame.
Also removes default implementations in GLObjectsProvider know that these methods have been implemented.
PiperOrigin-RevId: 517156304
Instead of using a long frameDurationUs with Math.floor, use a double
frameDurationUs with Math.round,
Before, playing an image with 30 fps over 1 second would result in the final
timestamp not being aligned to the expected 1 second timestamp. Over long
periods of time, this can lead to significant timestamp drift. Additionally,
for TimestampWrapper and constrained multi-asset, where TimestampWrapper
begins the 2nd effect on the 2nd asset's startTimeUs, this floor() behavior
can lead to the first few frames of the 2nd asset using the first asset's
effects, due to timestamps being mismatched.
PiperOrigin-RevId: 516529240
By making this method accept context, we can use DefaultDataSource.Factory in the DatasourceBitmapLoader to support a wider range on URI schemes in Bitmap Overlays. (and implement a local file picker for images for custom bitmap overlays in the demo transformer app)
PiperOrigin-RevId: 515013460
To create this file TextureInfo has been moved to common and renamed to GLTextureInfo.
We'll look to expand the interface in future to cover more of the methods around GL object maintenance in future as required.
PiperOrigin-RevId: 514445397
Before, we used to never call glDeleteFramebuffers, which could
in theory lead to leaks in the number of frame buffers
available and make releasing the GL context more expensive.
PiperOrigin-RevId: 514387847
Based on 1000 test runs an emulator, with the current timeout releasing
fails (even with no custom effects) about one percent of the time.
Releasing normally completes in about 30 ms but occasionally
`eglTerminate` took up to 200 ms (and even releasing an effect
took up to 80 ms in one case).
With the new timeout of 500 ms, we still catch stuck effects reasonably
quickly but the number of flaky test failures should be less than one in
ten thousand.
PiperOrigin-RevId: 512690715
Uses the first mediaItem's format as the output format.
If there is `Presentation` supplied in the `Composition.effects`, add it as the
last effect of the first EditedMediaItem.
PiperOrigin-RevId: 512082659
Also remove @WorkerThread annotations, as static checks associated with
this annotation aren't useful in this part of the codebase because
almost no methods are called on the main thread.
This change should be a no-op.
PiperOrigin-RevId: 512060367
Currently if releasing a shader program throws we don't release the GL
context, which could leak resources, and any errors are silently dropped
as we suppress notifications during releasing.
Improve resource cleanup and debuggability of errors from custom effects
by continuing to release shaders on failure (for runtime and
`VideoFrameProcessingException`s) and always clean up the GL context.
Note: this doesn't help with the case where releasing a custom shader
blocks for a long time, causing releasing the frame processor to
time out.
PiperOrigin-RevId: 512042501
Also, allow isNoOp to default to false without the TODO, so that implementations
of isNoOp must opt-in to implementing the override in order to be considered for
skipping the effect (ex. for transcoding in Transformer).
PiperOrigin-RevId: 511223540
Rename:
* MatrixShaderProgram to DefaultShaderProgram, and
* FinalMatrixShaderProgramWrapper to FinalShaderProgramWrapper.
PiperOrigin-RevId: 510498547
Rename ScaleToFitTransformation to ScaleAndRotateTransformation.
This better represents the operations that can be accomplished using this
effect. The name was originally named ScaleToFit* because it's not obvious how
to scale to fit using OpenGL, and this effect handled the scaling to fit in a way that no other MatrixTransformations did.
However, it's hard to discover how to rotate when skimming names of effects, so
it's probably more useful to convey that this effect rotates, than that it
scales to fit.
PiperOrigin-RevId: 510480078
Now that the GLEffectFrameProcessor handles external (video) and internal (image) input, components used only for external input needs should be moved to the ExternalTextureManager for code clarity
PiperOrigin-RevId: 509787494
It can be confusing how to use RgbMatrix for app devs not comfortable with image
manipulation, so add a helper class that simply translates rgb values the same
constant brightness value, to update the brightness values.
PiperOrigin-RevId: 509287229
GLEffectsFrameProcessor, MatrixShaderProgram and FinalMatrixShaderProgramWrapper are currently setup to handle the input frames coming from an external input (i.e. a video decoder). Image input is loaded into Bitmap objects at the start of the pipeline, so they are not produced externally. The changes provide a way for the frame processing pipeline to handle this "internal" (i.e. non-external) input.
PiperOrigin-RevId: 508645244
Based on [this conversation thread](https://chat.google.com/room/AAAA--f88ao/76Rem_cRCK8), I've opted to update the existing FrameProcessor.create() rather than deprecate it, as it is unlikely to be in use by apps outside google3.
PiperOrigin-RevId: 506920930
Set the correct output color on the debug SurfaceViewWrapper, so that SDR contents
can have an output transfer of either GAMMA_2_2 (Gamma 2.2) or SDR (SMPTE 170M).
This fixes an issue where in-app tone-mapping would output gamma 2.2, and the
SDR value incorrectly hardcoded here would lead to an error in the OpenGL, which
does not support SMPTE 170M.
PiperOrigin-RevId: 506684602
With the current ExtTexMgr,
it can happen that
- `x` frames are registered, but haven't arrived yet
- flush
- need to drop `x` frames when they arrive on SurfaceTexture
- status is reset to 0 pending, 0 available, drop `x` when frames arrive
- register one frame
- status is set to 1 pending, 0 available, drop `x` when frames arrive
- flush
- now the number of frame to drop is reset to `pending - available = 1`
- but it should be `x+1`
This CL solves the issue by reporting (by running the afterFlushTask) flush completes only after all the pending frames before calling flush are accounted for.
PiperOrigin-RevId: 506310671
Flushing resets all the texture processors within the `FrameProcessor`. This
includes:
- At the back, the FinalMatrixTextureProcessorWrapper, and its MatrixTextureProcessor
- At the front, the ExternalTextureManager
- All the texture processors in between
- All the ChainingGlTextureProcessorListeners in between texture processors
- All the internal states in the aforementioned components
The flush process follows the order, from `GlEffectsFrameProcessor.flush()`
1. Flush the `FrameProcessingTaskExecutor`, so that after it returns, all tasks queued before calling `flush()` completes
2. Post to `FrameProcessingTaskExecutor`, to flush the `FinalMatrixTextureProcessorWrapper`
3. Flushing the `FinalMatrixTextureProcessorWrapper` will propagate flushing through, via the `ChainingGlTextureProcessorListener`
Startblock:
has LGTM from christosts
and then
add reviewer andrewlewis
PiperOrigin-RevId: 506296469
The eotf is needed so that overlay (image) colors are correctly interpreted and mixed the linear video colors.
Also replaces the 100winners.png with "homemade" image file.
Added GlEffectsFrameProcessor test to justify that the color looks correct at the end of frame processing.
PiperOrigin-RevId: 506290309
For HLG input in transformer, FinalWrapper is configured to only output HLG to
encoder. But since DebugPreview is configured to take PQ for HDR content, the
color will not look correct.
This CL allows overriding the MatrixTP output transfer function, so
that FinalWrapper can output
- HLG to encoder
- PQ to debug preview
PiperOrigin-RevId: 506022840
Implementations outside media3 should be able to throw FrameProcessingException if they come across an error during configure().
PiperOrigin-RevId: 506020149
Tested adding 3, 15,16 different overlays in a single OverlayTextureProcessor. The program errored out when 16 overlays were added.
PiperOrigin-RevId: 504810157
This information may be needed for applications to create their own OpenGL contexts
and textures passed into or interacting with Effects.
PiperOrigin-RevId: 504342258
Based on experimentation, when CameraX is producing input for frame processor the buffer size is not being set to match the camera capture resolution, so the output has lower resolution than expected. Expose the default buffer size setter on `SurfaceTexture` to allow apps to process frames at full resolution for use cases like this one where the producer doesn't override the default buffer size.
PiperOrigin-RevId: 504022107
Also remove usages of TransformationRequest convenience methods
(setScale, setRotationDegrees and setResolution).
Some usages of setResolution can't be removed yet because they are used
for fallback.
PiperOrigin-RevId: 502415748
This is needed for constrained multi-asset to shift the timestamps of
the media items that are not the first in the sequence.
PiperOrigin-RevId: 502409923
Frame cache compensates for the fluctuation in frame processing times.
Imagine a frame takes 10ms to process, and the interval between two frames is
33ms. The third frame took 40ms to process.
If we don't have frame cache:
- Process frame 1, ready after 10ms, starts playback, now t=0 ms
- Start processing frame 2, ready at t=10ms,
- Release frame 2 at t=33ms
- We start processing the third frame at t=33ms
- The third frame is due presentation at t=66ms
- But frame 3 is available at t=73ms, late
If we have a frame cache of say 3 frams,
- Process frame 1, ready after 10ms, starts playback, now t=0 ms
- Start processing frame 2, ready at t=10ms
- Start processing frame 3, ready at t=50ms
- Release frame 2 at t=33ms
- Start frame 4, ready at t=60ms
- Frame 3 is due presentation at t=66ms
- Frame 3 isn't late
PiperOrigin-RevId: 501869948
Move BitmapTestUtil from media3.effect to media3.test.utils.
This allows this class to be accessed from both transformer and effect tests.
Refactoring change. No functional change intended.
PiperOrigin-RevId: 501837130
The migration strategy is to deprecate `androidx.media3.session.BitmapLoader` and copy the file into common since BitmapLoader is a public interface that apps could be relying on.
PiperOrigin-RevId: 501266521
Based on experimentation it seems that buffers can occasionally (roughly 1% of test runs) be dropped when rendering off-screen from EGL on the emulator. Specifically, in this test, sometimes after rendering three buffers with distinct timestamps only the first and third buffers' timestamps are handled in the `ImageReader`'s image available callback causing the assertion checking all frames rendered to fail. This behavior seems to be independent of the nanosecond presentation time attached to the buffers (as expected for off-screen rendering).
Introducing a pause of 1 second between rendering each frame reduces the flake rate to around 1/2000. This increases the run time of some of the tests, so this change also removes the 5 second `FRAME_PROCESSING_WAIT_MS` (it seems to be unnecessary when rendering off-screen) and instead uses a latch to wait until the frame processor has handled 'end of stream'.
PiperOrigin-RevId: 499440591
This CL replaces concurrent collections and atomic primitives with a single
lock, this way the code is easier to reason about.
PiperOrigin-RevId: 496718057
FrameProcessor already support using different transfer function for input and
output color. This CL has two major changes:
- Create an eglSurface that recognizes BT.2020 PQ
- This requires a separate extension that works only after 33
- So we current throw, if input is HDR, and this extension doesn't work
- Create FrameProcessor with PQ output transfer function
PiperOrigin-RevId: 496023758
These terms are easier to understand, and make sense in the context of
MatrixTextureProcessor now that a MatrixTextureProcessor may have a
different input and output transfer.
PiperOrigin-RevId: 493265980
Modify FrameProcessor and MatrixTextureProcessor interfaces to support
different input and output color transfers. Does not implement conversion between
color ranges (ex. HDR and SDR), but should allow for conversion between color
transfers of the same color range (ex. HLG and PQ).
This supports in-app tone mapping, where we need a single FrameProcessor to
input HDR color transfers (ex. HLG/PQ) and output SDR (ex. gamma2.2). This also
supports previewing, where we need a single FrameProcessor to be able to input HLG
and output PQ.
Manually tested by confirming colors still look right on SDR and HDR videos
with a rotation and color affect applied.
PiperOrigin-RevId: 493108678
From this CL on, FrameProcessor listeners will be invoked from an Executor that
is passed in when creating the FrameProcessor.
GlTextureProcessor needs to invoke the ErrorListener on the said Executor too.
PiperOrigin-RevId: 493018583
Previously, FrameProcessor never had the usecase in which the output surface
is replaced, while previewing introduced this usecase.
When switching output surfaces, we need to destroy the EGL Surface linked to the
surface that is being swapped out, because an EGL surface is linked to the EGL
display (which is not destroyed even when releasing FrameProcessor).
A GL exception will be thrown in the following scenario if we don't destroy the
EGL surface:
1. Creates a Surface, the surface is identified by address 0x11
2. Sets Surface(0x11) on FrameProcessor. Eventually an EGL surface is created
to wrap Surface(0x11)
3. Release FrameProcess, this releases the EGL context
4. Instantiate a new FrameProcessor, sets Surface(0x11) as the output
5. When FrameProcessor creates an EGL surface to wrap Surface(0x11), GL throws
an exception, becasue Surface(0x11) has previouly been connected to an EGL
surface.
PiperOrigin-RevId: 489590072
`availableFrameCount` tracks the number of frames that is avilable on the
`SurfaceTexture`, but haven't been used (by `updateTexImage()`) yet. Thus
semantically this counter should only be decremented after calling
`updateTexImage()`, not before it.
Also reworded `getPendingFrameCount()` javadoc, "external texture" is an
internal state that is not publicised anywhere.
PiperOrigin-RevId: 488765174
Public methods may only refer to public types in their signature. This
change ensures that by switching to a public supertype everywhere.
PiperOrigin-RevId: 485568625
Make it easier to support use of RGBA_101012 rather than RGBA_8888 for EGL
contexts, displays, and surfaces.
This tangentially supports adding HDR tests, by slightly simplifying the color
selection logic we'd have to add in HDR tests.
PiperOrigin-RevId: 482219428
