In follow-ups the FrameProcessorChain will set an instance of this
listener for each GlTextureProcessor to chain it with its previous
and next GlTextureProcesssor.
PiperOrigin-RevId: 455628942
(cherry picked from commit 981baae709)
The wrapper
* catches exceptions for each task and notifies the
listener (this will be used more in follow-ups when processFrame
is split into lots of listeners and callbacks),
* removes finished tasks from the queue and signals any exceptions
that occurred to the listener each time a new task is executed.
PiperOrigin-RevId: 455345184
(cherry picked from commit ee847d92c5)
This change adds a SurfaceProvider interface which is necessary to
allow for texture processors whose output size becomes available
asynchronously in follow-ups.
VTSP's implementation of this interface wraps the encoder and provides
its input surface together with the output frame width, height, and
orientation as used for encoder configuration.
The FrameProcessorChain converts the output frames to the provided
orientation and resolution using a ScaleToFitTransformation and
Presentation replacing EncoderCompatibilityTransformation.
PiperOrigin-RevId: 455112598
(cherry picked from commit ea7f1ca1e3)
Based on
https://developer.android.com/reference/android/media/MediaCodec#using-an-output-surface,
frame dropping behaviour depends on the target SDK version.
After this change transformer will only use
MediaFormat#KEY_ALLOW_FRAME_DROP if both the target and system SDK
version are at least 29 and default to its pre 29 behaviour where each
decoder output frame must be processed before a new one is rendered
to prevent frame dropping otherwise.
Also remove deprecated Transformer.Builder constructor without a
context and the context setter.
PiperOrigin-RevId: 453971097
(cherry picked from commit a105d033a7)
Transformer always enabled glAssertionsEnabled, so there should
be no functional change.
ExoPlayer previously disabled glAssertionsEnabled, so GlUtil logged
GlExceptions instead of throwing them. The GlExceptions are now
caught and logged by the callers so that there should also be no
functional change overall.
This change also replaces EGLSurfaceTexture#GlException with
GlUtil#GlException.
PiperOrigin-RevId: 453963741
(cherry picked from commit cc1f32d094)
This removes the prior restriction of needing to remember not to crop and set aspect ratio in the same Presentation.Builder, and makes each class a bit more targeted.
This is partially made feasible by the past work to merge consecutive
MatrixTransformations into a single MatrixTransformationFrameProcessor, which
ensures that there's no loss in quality between successive MatrixTransformations.
PiperOrigin-RevId: 453660582
(cherry picked from commit c5b881e089)
SingleFrameGlTextureProcessor is now an abstract class containing a
default implementation of the more flexible GlTextureProcessor interface
while still exposing the same simple abstract methods for single frame
processing it previously did.
FrameProcessorChain and GlEffect will be changed to use
GlTextureProcessor in follow-ups.
PiperOrigin-RevId: 453633000
(cherry picked from commit 457f446114)
Implementations of this interface will be able to drop or add frames,
change timestamps, accept multiple input frames before producing
output, and process frames on their own background thread.
A default implementation of this interface will be added to SingleFrameGlTextureProcessor in a follow-up.
PiperOrigin-RevId: 453159835
(cherry picked from commit 63436390de)
This internal listener avoids wrapping the TransformationExceptions
in PlaybackExceptions that are handled via the Player.Listener and
is also used for FrameProcessingExceptions which already avoided
the PlaybackException layer previously.
This listener will also be useful in follow-ups for encoder-related
TransformationExceptions that are thrown in the SurfaceProvider that
will be called on the GL thread.
PiperOrigin-RevId: 452074575
(cherry picked from commit 35b5147eb1)
Once the more advanced GlTextureProcessor interface exists,
it will be possible to change the output size of a GlTextureProcessor
between frames. To keep the re-configuration based on the frame sizes
minimal, things indepedent of the frame size, such as the GlProgram,
can be initialized in the constructor.
PiperOrigin-RevId: 451997584
(cherry picked from commit 87ab96d352)
ExoPlayer applies a large time offset to buffers so that, if the input has negative timestamps, generally buffers seen by the decoders should have positive timestamps. Modify how the offset is handled in `Transformer` so that decoders and encoders generally see positive timestamps, by leaving the offset on samples when reading them in the base renderer (remove the code that removed the offset), and then removing the offset when muxing. Also update the frame processor chain and slow motion flattening code to retain the existing behavior after this change (these both need original media presentation timestamps)
Tested via existing end-to-end tests and manually verified that the overlay frame processor shows the expected original media timestamps.
Aside: we don't need the same logic as ExoPlayer to track stream offsets across the decoder yet, because we don't try to handle stream changes during playback in single asset editing. (There is an edge case of multi-period DASH that may not work but I doubt anyone will use that as input to `Transformer` before we change the code to handle multi-asset properly.) In future we should try to refactor interaction with the decoder to use the same code for Transformer and ExoPlayer.
PiperOrigin-RevId: 451846055
SSIM calculation requires the input and output dimensions to be identical.
For devices that can't encode the input dimensions, skip SSIM calculations and
log the cause. Only apply this on tests where the encoder may not support the
input file dimensions.
PiperOrigin-RevId: 451364904
Decode-only video frames (needed when the frame at / first frame after the
clipping start is not a key frame) need to be decoded but not passed to
the frame processor chain or encoder.
The clipping start offset needs to be removed from the frame timestamps
in the passthrough and video pipelines.
There are no changes needed for this in the audio pipeline, as it doesn't
use the input timestamps -- it uses its own timestamps derived from the
buffer sizes instead.
Also add demo option to try this out.
#minor-release
PiperOrigin-RevId: 451353609
Also update names of implementations to match design doc.
In follow-ups, SingleFrameGlTextureProcessor will become
an abstract implementation of a new GlTextureProcessor
interface.
Texture processor makes sense as it processes OpenGL textures.
The term frame processor will be used for something else in
follow-ups.
PiperOrigin-RevId: 451142085
The debug surface view's output surface can become invalid during a transformation due to the parent activity pausing, for example. This can currently cause a crash when backing out of the `TransformerActivity` in the demo because the surface can be destroyed before the transformer has fully canceled.
Also clarify naming of the outputSurface and inline the private method that created `EGLSurface`s (it was shorter after removing the debug preview).
PiperOrigin-RevId: 449963440
The old reference was just for prototyping HDR. The new reference is for planned work to use the correct formats for input and output for HDR editing in GL.
PiperOrigin-RevId: 449211792
When using a MatrixTransformationFrameProcessor per transformation
matrix, each frame processor's shader applies the matrix to the
vertices and clips the result to the NDC range when drawing the
output frame.
This change combines consecutive MatrixTransformations into a single
MatrixTransformationFrameProcessor by multiplying the individual
matrices while updating and clipping the visible polygon after
each matrix and mapping the resulting visible polygon back to the
input space so that its vertices and the combined transformation
matrix can be used in the shader.
PiperOrigin-RevId: 448521068
This listener replaces
FrameProcessorChain#getAndRethrowBackgroundExceptions.
The listener uses a new exception type FrameProcessingException
separate from TransformationException as the frame processing
components will be made reusable outside of transformer soon.
PiperOrigin-RevId: 447455746
This constant is used for https://docs.gl/es2/glVertexAttribPointer
which takes the number of components per generic vertex attribute
(meaning the size of the individual coordinate vectors here) not the
number of attributes (the number of vertices that the old constant
name referred to).
PiperOrigin-RevId: 447427241
We won't try to provide/rethrow helpful error messages that are already
provided by GL, as this sort of task would expand into writing a GL verifier.
A Gl verifier is unnecessarily complex for minimal payoff, especially as Apps
expected to read GL error messages would mostly be those writing custom
GlFrameProcessors, who should be already be familiar with reading GL error
messages anyways.
PiperOrigin-RevId: 446950837
ScaleToFitFrameProcessor, PresentationFrameProcessor,
and EncoderCompatibilityFrameProcessor now each implement
MatrixTransformation instead of wrapping
MatrixTransformationFrameProcessor.
PiperOrigin-RevId: 446480286
This change splits AdvancedFrameProcessor into 4 files:
- MatrixTransformationFrameProcessor for the GlFrameProcessor
implementation
- MatrixTransformation and GlMatrixTransformation for the GlEffect
specification
- MatrixUtils for the static matrix helpers
PiperOrigin-RevId: 446236384
Also add warning for what can happen if this is ignored.
Previously the return value was overridden by setOutputSurface, now
setOutputSurface only overrides the values using internally.
PiperOrigin-RevId: 445377036
To ensure frame processor operations operate on square pixels,
make the frame taller or wider for non-square input pixels.
In addition to automated tests, this was tested by changing the
inputFormat.pixelWidthHeightRatio in the TransformerVideoRenderer.
PiperOrigin-RevId: 444553517
The last frame processor could use a different drawing command
than a four vertex triangle strip. So we need call its drawing
method again instead of assuming the shape.
PiperOrigin-RevId: 444217274
Avoid no-op FrameProcessors in VideoTranscodingSamplePipeline, to avoid
creating resources for and executing GL for a no-op vertex+fragment shader.
EncoderCompatibilityFrameProcessor will still always be exercised.
PiperOrigin-RevId: 443675833
Split rotationDegrees changes to EncoderCompatibilityFrameProcessor, a new
FrameProcessor.
This removes automatic rotationDegrees adjustments from Presentation, which
allows Presentation to be used for changes before the end of a
FrameProcessorChain pipeline.
PiperOrigin-RevId: 443387226
The issue with decoding getting stuck is still reproducible on `samsung/beyond1lteeea/beyond1:12/SP1A.210812.016/G973FXXUEGULB:user/release-keys`.
PiperOrigin-RevId: 443368546
Some devices under report their resolution support, like 2144 for 2160 in
H265, 1072 for 1080 in H264. This CL only takes care of these two cases,
- reporting 1920x1080 is supported when the device reports 1920x1072, and
- reporting 3840x2160 is supported when the device reports 3840x2144
PiperOrigin-RevId: 443095042
This is safer because it will prevent any future problems with
creating a new FrameProcessorChain before the previous one has
completed its async release.
From [eglDestroyContext documentation](https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglDestroyContext.xhtml):
"If the EGL rendering context is not current to any thread,
eglDestroyContext destroys it immediately."
The context isn't current to any thread here because GlUtil calls
eglMakeCurrent with EGL_NO_CONTEXT before calling eglDestroyContext.
So everthing should be released once the FrameProcessorChain's
release task terminates.
PiperOrigin-RevId: 442807484
Scaling and rotation using ScaleToFitFrameProcessor may change the
the resolution and aspect ratio, so defaulting to the same as the
input is no longer accurate.
PiperOrigin-RevId: 441463349
What a minimal implementation should include is now explained in the
interface javadoc while the method name reflects what the method does.
PiperOrigin-RevId: 441432059
