This requires an additional nanos to micros conversion because
the SurfaceTexture uses nanos. But as the timestamps from the
MediaCodec decoder (propagated in DefaultCodec#releaseOutputBuffer) are
in microseconds no precision is lost here.
Also add test that checks output video duration.
PiperOrigin-RevId: 438010490
MAXIMUM_AVERAGE_PIXEL_ABSOLUTE_DIFFERENCE was copied from a test
class, but BitmapTestUtil isn't a test. So the javadoc needs
rewording to reflect that.
PiperOrigin-RevId: 438001833
From Android T onwards `MediaCodec` supports requesting tone-mapping down to
SDR. Add an option to request this behavior and document that it isn't
supported before T. Also add an option in the demo app to try it out.
Tested manually on a prerelease build.
PiperOrigin-RevId: 437765325
The thread name is used to verify the thread in both createOpenGlObjectsAndInitializeFrameProcessors() and processFrame().
Also remove glThread field that was only used for this verification.
PiperOrigin-RevId: 437730804
Since the output size can be overridden, the viewport should be
ouputWidth/Height and NOT the ExternalCopyFrameProcessor's output size
which matches the input size.
PiperOrigin-RevId: 437256635
* The AdvancedFrameProcessor calls use() in updateProgramAndDraw().
* The AdvancedFrameProcessor has the same input and output dimensions.
PiperOrigin-RevId: 437231350
On some devices, decoding gets stuck when the number of frames pending at the
`SurfaceTexture` is too high. We added a workaround that only allows one frame
to be pending at a time. That fixed the issue, however, based on on-device
testing it seems that it's safe to queue more than one frame.
Add a method that returns a safe estimate of the number of frames that can be
pending at a time, and use this to limit the number of frames that can be
released from the decoder but not processed by the frame processor chain.
PiperOrigin-RevId: 437057075
Configuring the frame sizes between frame processors is now the
FrameProcessorChain's rather than the caller's responsibility.
The caller can getOutputSize() and override it for encoder fallback
in configure().
PiperOrigin-RevId: 437048436
The factory method is replaced by a public constructor and
configure() method which configures the input/output surfaces
and handles the OpenGL setup.
This is a prerequisite for removing the responsibility of the
caller to configureSizes() before creating the chain in a follow-up.
PiperOrigin-RevId: 437028882
This allows us to bypass many device-specific issues, that only occur when
decoding directly to an encoder surface, without OpenGL. This also allows us
to maintain fewer code branches, which require additional testing to verify
correctness.
PiperOrigin-RevId: 437003138
Allow apps to modify how frames are presented, via modifying resolution.
A follow-up CL will provide aspect ratio, cropping, etc.
PiperOrigin-RevId: 436963312
SEF is similar to frame editing, where the input and output videos are
not intended to match, so SSIM will not provide a reliable value. To
check SSIM correctly in future, we would need to provide
golden/expected video files to compare output against.
PiperOrigin-RevId: 436707240
The FrameProcessorChain manages a List<GlFrameProcessor>.
FrameProcessorChainDataProcessingTest now tests chaining ScaleToFit-
and AdvancedFrameProcessors.
PiperOrigin-RevId: 436468037
This currently causes the test to fail on Pixel 6 Pro running a recent S build
SQ1D.220205.004.
There is no need to test audio transcoding while we are measuring video
quality.
PiperOrigin-RevId: 435635314
ExternalCopyFrameProcessor's output dimensions match the input
size not the output size. So the intermediate texture size
should match the input size.
Also rename configureOutputDimensions to configureOutputSize.
PiperOrigin-RevId: 435058789
Use android.util.Size, whose naming is much easier to understand than Pair<Integer, Integer>, in both FrameProcessor and EncoderUtil.
PiperOrigin-RevId: 434813986
* Move auto-adjustments for transformation matrices from the
VideoTranscodingSamplePipeline constructor to the new
ScaleToFitFrameProcessor.
* Add GlFrameProcessor#getOutputDimensions() to allow for GlFrameProcessors with
different input and output dimensions. This is a prerequisite for
Presentation.
* Tested with unit tests (and manually just in case).
* A follow up CL will implement change the FrameProcessor input to be scale and
rotate values as requested by the user. This was kept out of this CL to
reduce CL review size. Presentation will also be implemented in a follow up
CL.
PiperOrigin-RevId: 434774854
As pointed out in a previous review, createFallbackTransformationRequest
can be a bit confusing to parse.
Added javadocs and renamed parameters appropriately, to make it slightly
more easy to understand.
PiperOrigin-RevId: 434733313
The variable marks the coordinates used to sample from a texture, so hopefully
this makes the naming a bit more descriptive.
This renames vTexCoords and aTexCoords. No functional changes intended.
PiperOrigin-RevId: 433499934
If an OpenGL call blocks because the encoder's input surface is full,
this will now block the background thread while the main thread can
continue querying encoder output and free up encoder capacity until
it accepts more input unblocking the background thread.
PiperOrigin-RevId: 433283287
With the new version, we try the following before fixing resolution:
- Fix size alignment
- Try 3/4 the width and height
- Try 2/3 the width and height
- Try 1/2 the width and height
Also: align the resolution ends in 1 or 9 to 0.
PiperOrigin-RevId: 433206358
Previously, we've used getSupportedHeights/Widths() to find the supported
resolution. However, the height/width can be over-reported when using these
APIs. For example, getSupportedWidths and getSupportedHeights can both return
3840, but the supported height when using 3840 as width is only 2160.
PiperOrigin-RevId: 432926192
This will allow for easier customisation of the additional tasks
performed by the test runner, such as calculating metrics like SSIM.
PiperOrigin-RevId: 432434850
This test tests the same cases as the FrameEditorDataProcessingTest
as currently the main FrameEditor functionality is to apply a
transformation matrix using a TransformationFrameProcessor.
PiperOrigin-RevId: 431642066
We use SSIM to measure the transcoding quality between. SSIM is a widely used
tool that compares the luma channel between two images, and generates a score
from 0 to 1 that indicates "how similar" the two images are.
In `SsimHelper`, we decode the two videos, extract matching frames and
calculates the mean SSIM (SSIM averaged all matching frames) for both videos.
Matching frames are referred to as "comparisonFrame" in the CL, which is
selected based on the frame number and a user-set comparison interval.
For instance, if the interval is 7, then every seventh frames are compared.
We use MediaCodec/MediaExtractor to decode the video, and use ImageReader to
extract the decoded frame.
The SSIM calculation logic is a inspired by and modified from the CTS
[MSSIMMatcher](https://cs.android.com/android/platform/superproject/+/master:cts/tests/tests/uirendering/src/android/uirendering/cts/bitmapcomparers/MSSIMComparer.java;l=1?q=mssimcom)
that has some errors and extra features we don't need (like handling RGB
images).
Adds TranscodeQualityTest to ensure high quality transcoding.
PiperOrigin-RevId: 430951206
We need to access internal state to work around resources not being released on
old API versions. Add a reference to the bug about this and suppress the lint
warning.
#minor-release
PiperOrigin-RevId: 430190794
All (later customizable) GlFrameProcessors after the
ExternalCopyFrameProcessor receive their input from a normal OpenGL
texture not an external texture, so they won't need to worry about
the textureTransformMatrix.
PiperOrigin-RevId: 430165652
There are two major blockers to this test:
- H265 muxing is not available for API<24, so setting video mimeType to H265
will fail on those devices.
- AMR audio encoding is buggy on some device and it's not a widely used format.
The solution: use a video that is encoded with AVC/MP3, to ensure transcoding
to AVC/AAC.
PiperOrigin-RevId: 429648598
Re-enable tests that have no muxer support for timestamps going backwards.
Tests running on the B-frame sample will be added in a future commit.
#mse-bug-week
PiperOrigin-RevId: 429599177
We use the `createForCodec` method that does not take a `MediaFormat` during
transformation, the error message always includes "no configured MediaFormat",
which is false.
PiperOrigin-RevId: 429553573
Calling `MediaMuxer.writeSampleData` can block indefinitely on old API versions. It is better not to call this method to fail quickly with an exception rather than getting stuck.
Based on on-device testing media muxer doesn't generally handle out of order samples before API 25. There are a small number of devices where this does succeed but it seems preferable to turn this off everywhere to keep the code simple and have consistent behavior. Once we switch to in-app muxing this limitation will no longer apply.
#mse-bug-week
PiperOrigin-RevId: 429070255
Tested:
Verified that the additional information is available through
instrumentation tests, as well as via manual testing.
#mse-bug-week
PiperOrigin-RevId: 429038695
- The resources were released twice before, which is not necessary since
the MSG_RELEASE message is already in the internal player queue.
- The demo app was failing because the stop watch was stopped in
onTransformationError after being reset.
#minor-release
#mse-bug-week
PiperOrigin-RevId: 428794426
The GlFrameProcessor handles everything related to the GLSL program,
the FrameEditor manages the GL context and the data flow including
the input SurfaceTexture and output EGLSurface.
This will be split up further in follow-up CLs so that
GlFrameProcessors can be chained. At this CL, there are no
functional changes intended.
PiperOrigin-RevId: 428779179
We have seen devices running on API21/23 fail transcoding because of setting
encoding profile/level.
Some devices (ale-123/nexus 7) on API21 returns ENOSYS (Function not
implemented) when being configured with a profile setting. (although API21
introduced the capability of setting encoding profile)
Some devices (nexus 5) on API23 fails configuration with a specific parameter
set, despite advertising support for it.
Not setting the baseline profile has no effect on encoding, because when not
set, the encoding will pick a suitable profile to use. Since baseline is
the lowest possible profile, the auto-picked value can't be worse than
baseline.
Ref: b/218696352
PiperOrigin-RevId: 427792124
On some Android devices, the return value of
```
MediaCodecInfo.getCapabilitiesForType(mimeType).profileLevels
```
contains one entry for each encoding profile, like <profile, maxSupportedLevel>
but on some other devices, there are multiple entries for the same profile,
like <HIGH_PROFILE, LEVEL1>, <HIGH_PROFILE, LEVEL2>, <HIGH_PROFILE, LEVEL3>,
where we need to iterate through all the entries and find the max.
PiperOrigin-RevId: 427727030
On some old devices, the encoding level needs to be set with the encoding
profile, but not on newer devices.
The profile/level override is applied by following
https://developer.android.com/guide/topics/media/sharing-video
PiperOrigin-RevId: 427008536
This is a breaking change if the annotation itself is in use in Kotlin
code. It's judged that the IntDefs in this commit are unlikely to be
referred to often in Kotlin code. This is because they're either:
- Related to esoteric parts of the library, or
- In a common part of the library but only returned from methods (and
never passed to callback methods).
A follow-up change will fix the positions of existing usages to match
this new config.
#minor-release
PiperOrigin-RevId: 426410237
Fallback is only disabled for robolectric and instrumentation tests.
For MH tests, fallback is not disabled, as it may be needed due to
the broad range of devices available.
PiperOrigin-RevId: 426403167
Reason for not rolling back the rollback 9151103968: file name changed and
file content moved, the automated tool is unable to correctly apply the change.
Apply suggested AVC profile depending on the API version.
Use `AVCProfileHigh` only when there's encoder support.
PiperOrigin-RevId: 426363780
This change makes GlUtil.Program an outer class named GlProgram,
and also moves private static helpers as well as the inner classes
Attribute and Uniform which were only used by GlUtil.Program to
GlProgram. Other static utility methods remain in GlUtil.
No functional changes intended.
PiperOrigin-RevId: 426119299
The longer list of targets is only necessary for backwards
compatibility with existing Kotlin code that will stop compiling
if the position of the annotation becomes 'wrong' by marking it only
TYPE_USE. Since none of these IntDefs have been released (except in
media3 alpha1) we don't need to maintain this compatibility.
Also add a comment to all the places that *do* need the longer list of
targets, in order to explain why it's there and discourage copy-pasting
when defining new IntDefs in future.
Also fix some single-element arrays to remove the array notation.
#minor-release
PiperOrigin-RevId: 426108537
- The MIME type should ideally default to HEVC if there is an encoder for it.
- Next, check if AVC is supported.
- If there is no encoder for AVC, then we should pick an encoder in the list of
existing encoders instead of abandoning the transformation.
PiperOrigin-RevId: 425900638
Tested by confirming transformations still work and write to a output file in a
scoped-storage directory on a:
* Nexus 6P API 23 emulator
* Google Pixel 4 API 31 physical device
PiperOrigin-RevId: 425644266
- Add a checkbox in the demo app to enable experimental HDR editing.
- Add an `experimental_` method to `TransformationRequest` to enable HDR editing.
- Add fragment/vertex shaders for the experimental HDR pipeline. The main difference compared to the existing shaders is that we sample from the decoder in YUV rather than RGB (because the YUV -> RGB conversion in the graphics driver is not precisely defined, so we need to do this to get consistent results), which requires the use of ES 3, and then do a crude YUV -> RGB conversion in the shader (ignoring the input color primaries for now).
- When HDR editing is enabled, we force using `FrameEditor` (no passthrough) to avoid the need to select another edit operation, and use the new shaders. The `EGLContext` and `EGLSurface` also need to be set up differently for this path.
PiperOrigin-RevId: 425570639
TransformerTest sounds like a unit test for Transformer but these
tests test behaviour that involves multiple stages of the pipeline.
PiperOrigin-RevId: 425378369
If
a) the end of stream buffer arrives with a frame rather than an
empty buffer or
b) processDataV29() renders several decoder output buffers to the
FrameEditor's input Surface immediately before encountering the
EOS flag
these frames were previously stuck in the FrameEditor's input Surface
and never fed to the encoder.
PiperOrigin-RevId: 424898820
This test tests that all frames are processed when transcoding
video to a different sample MIME type (and that the transformation
completes successfully).
PiperOrigin-RevId: 424896014
If muxerWrapper.release() was throwing an exception, the progress state
was not updated and getProgress could throw an exception.
#minor-release
PiperOrigin-RevId: 424696783
When the decoder output buffer was partially read, a call to
Codec.getOutputBuffer() was returning the same buffer, but with the
position reset to 0. The reason was that, in
Codec.maybeDequeueAndSetOutputBuffer(), mediaCodec.getOutputBuffer()
was called with the same buffer index (L350 in old rev), even though
there was already a buffer available (outputBufferIndex >=0). This
change avoids calling mediaCodec.getOutputBuffer() if the previous
buffer has not been released.
#minor-release
PiperOrigin-RevId: 424612197
If the encoder picks a fallback resolution the video pipeline needs
to take this into account when configuring the frameEditor and when
setting up the fallback TransformationRequest that's passed to the
fallbackListener.
PiperOrigin-RevId: 424611290
If the output sample MIME type is inferred from the input
but is not supported by the muxer, we fallback to transcoding
to a supported sample MIME type.
The audio and video renderers need to make sure not to select the PassthroughSamplePipeline for this case. Which sample MIME type
to choose is decided by the EncoderFactory.
PiperOrigin-RevId: 423272812
We may fall back to a different sample MIME type because
a) the sample MIME type inferred from the input is not supported
by the muxer or b) no encoders are available for the the requested
sample MIME type.
PiperOrigin-RevId: 422849036
TransformationRequest is otherwise immutable, so if we modify the
transformationMatrix in place (done before this cl) this may cause
confusing behaviour for apps when they reuse a TransformationRequest.
PiperOrigin-RevId: 422822916
After implementing fallback, it won't always be possible to
differentiate between muxer and encoder as the cause of an output
format not being supported.
PiperOrigin-RevId: 422780443
This CL implements fixing the input format to the encoder spec. Fixed
parameters include:
- MIME type
- Profile & level
- Resolution
- frame rate, and
- bitrate
PiperOrigin-RevId: 422513738
The app will be notified about fallback using a callback on
Transformer.Listener. Fallback may be applied separately for
the audio and video options, so an intermediate internal
FallbackListener is needed to accumulate and merge the track-specific
changes to the TransformationRequest.
PiperOrigin-RevId: 421839991
Simplifying and clarifying variables, and adding comments.
Tested by confirming demo-gl and demo-transformer both
correctly display videos
PiperOrigin-RevId: 421792079
After this change exceptions throw by MediaCodec during
encoding/decoding will result in TransformationExceptions with
ERROR_CODE_ENCODING_FAILED/ERROR_CODE_DECODING_FAILED.
Before this change ERROR_CODE_FAILED_RUNTIME_CHECK was used.
PiperOrigin-RevId: 421560396
Expected images are taken on emulators, so a larger acceptable
difference from expected images must be accepted on physical devices.
PiperOrigin-RevId: 421543441
Codec and its factories can use MediaCodec directly as for API >= 21,
the SynchronousMediaCodecAdapter methods used in Codec just correspond
to a single MediaCodec call each so there is no reason to have another
wrapping layer.
PiperOrigin-RevId: 421041177
Exceptions thrown by MediaMuxer are converted MuxerExceptions
and later to TransformationExceptions with ERROR_CODE_MUXING_FAILED.
PiperOrigin-RevId: 421033721
The encoder and sonic are now set up in the constructor rather
than in a configuration method called from processData(). This
is more similar to VideoSamplePipeline and reduces null checks.
PiperOrigin-RevId: 420260526
Only allowing TransformationExceptions to be created using the factory methods helps keeping error messages consistent. This is consistent with ExoPlaybackException.
PiperOrigin-RevId: 419841025
Compensate for aspect ratio of input frames, so that they're applied on
rectangular frames instead of square normalized-device-coordinate frames.
This fixes distortion most visible when rotating any GL video 45°
(non-rectangular frames) or 90° (stretched frames)
Tested by rotating several landscape/portrait demo videos.
(Automated tests will follow in <unknown commit>)
PiperOrigin-RevId: 419619743
* Move checking that the output format is supported by the muxer
from supportsFormat (which deals with the input format) to
ensureConfigured.
* Add maps for the supported MIME types so that the muxer can
return what MIME types it supports rather than just check a
MIME type.
PiperOrigin-RevId: 419578165
This is better than silently dropping tracks as done previously. Later,
we will implement fallback to transcoding to a supported MIME type.
PiperOrigin-RevId: 418006258
This more closely matches the pattern we have for all implementations
except DefaultMediaSourceFactory (e.g. ProgressiveMediaSource.Factory)
and other factory interfaces like (Http)DataSource.Factory.
PiperOrigin-RevId: 417826803
Switch to using sentence-case naming convention but with one character prefixes for different types.
This is a no-op change.
PiperOrigin-RevId: 417791624
Move static factories into a separate class and make it implement an interface
that will let tests customize encoder/decoder creation.
PiperOrigin-RevId: 417610825
Transformer uses ExoPlayer for reading input. Apps using Transformer
do not need to know this. So, PlaybackExceptions are converted to
TransformationExceptions with the same message, cause and error code.
The corresponding IO error codes are copied from PlaybackException.
PiperOrigin-RevId: 416793741
UnsupportedEglVersionException() is only used once, and seems a bit too
specific for Transformer. Also, it's possible for eglCreateContext to fail for
other reasons besides lack of support, so it wasn't always accurate when
thrown.
It is possible for devices not to support EGL version 2.0 though, per
https://source.android.com/devices/graphics/implement-opengl-es, which doesn't
specify the EGL version that must be supported.
PiperOrigin-RevId: 415489396
tl;dr:
In the previous transformer, the transcoding flow is
- If a the GL's input surface (from decoder) does not have data, wait 10ms
(DO_SOME_WORK)
- Else, make the decoder render **ONE** frame to the GL's input surface
- Wait at least 10ms, until the frame's texture is available
- Then process the texture
The process is quite slow, so in the new version, we do:
- If a the GL's input surface (from decoder) does not have data, wait 10ms
(DO_SOME_WORK) **same**
- Else, make the decoder render **as many frames** to the GL's input surface
- Process **as many** available textures in this DO_SOME_WORK cycle
PiperOrigin-RevId: 415474722
The input rotation is used to rotate the video during decoding, the
video is rotated so that it is in landscape orientation before encoding
and a rotation is added to the output format where necessary so that
the output video has the same orientation as the input.
PiperOrigin-RevId: 415301328
Being able to see the output of the GL pipeline is useful for debugging. For
example, when we previously saw flakiness it would have been useful to be able
to tell quickly whether the output looked wrong without needing to run a
transformation to the end then inspect the output file, and when working on
support for HDR editing it's useful to be able to do manual testing on devices
that don't support HDR encoding (but do support decoding/processing it with
GL).
Also change the progress indicator to be linear as this looks better in the
demo app when shown next to the debug preview.
PiperOrigin-RevId: 414999491
Also, add 144p as an acceptable output resolution, to allow for
a more obvious resolution difference when running the demo.
PiperOrigin-RevId: 414406664
outputHeight is the actual output height while
transformation.outputHeight could be Format.NO_VALUE
causing the FrameEditor to be used more often than
necessary in the old version.
PiperOrigin-RevId: 414304251
When calling Android's Log class directly, there's a LongLogTag
lint check that detects tags over the 23 char limit, however it
cannot detect long log tags in ExoPlayer due to the way that we
log via our own Log class. This commit adds @Size annotations to
enforce the same rule.
PiperOrigin-RevId: 413976364
Increase timeout for dequeueing a frame from the codec to reduce flakiness. At a timeout of 2 seconds there was a 2/1000 flake rate and at 3 seconds 0/1000. Set the timeout to 5 seconds to give plenty of leeway.
PiperOrigin-RevId: 413946915
Previously, transformation_matrix was incorrectly applied to
texture sampling coordinates, which led to transformations
seemingly moving in the opposite position, and an undesirable
GL_CLAMP_TO_EDGE behavior when sampling outside the edge of
the texture.
PiperOrigin-RevId: 413653360
It seems fine to remove the documentation about the WebM case now we are only supporting unfragmented MP4, so that new users coming to this API aren't confused about how to set the container MIME type.
PiperOrigin-RevId: 413611472
Allows a transformation matrix to be input into Transformer,
to apply vertex transformations like cropping, rotation,
and other transformations built into android.graphics.Matrix.
Not building out into a VertexTransformation class yet, as
that class structure wouldn't make sense until we can modify
resolution, per TODOs.
PiperOrigin-RevId: 413384409
This will remove the need to implement compat code handling very old API
versions where some symbols are not available, and it reduces the burden of
dealing with media framework issues around concurrent codec usage that are
worse on older API versions. Top apps that we've surveyed as potential users
for transformer library features are using API 21 or later.
PiperOrigin-RevId: 413341540
Sometimes the empty end of stream buffer has a non-zero
data limit. Calling flip first, resets the limit to the
position which is zero in these cases.
PiperOrigin-RevId: 413156455
The test extracts and decodes the first video frame in the test media, renders it to the frame editor's input surface and then processes data. It then reads back the output from the frame editor, converts it to a bitmap and then compares that with a 'golden' bitmap (which is just the same as the test media's first video frame).
PiperOrigin-RevId: 413131811
*** Original commit ***
Remove usage of @ForOverride.
Fixes the gradle compilation failures.
Gradle dependencies need revising if we want to be using this, as
checkerframework is ahead of their latest version, such that we
can't compile.
***
PiperOrigin-RevId: 412901827
- This format is passed to the PassthroughPipeline, which doesn't use
any decoder.
- In most other cases where it is used, it is not relevant that this
format will be or has been passed to the decoder. What's relevant is
that it is the format of the input.
PiperOrigin-RevId: 412093371
We need the filename of the output videos to be predictable, because
MobileHarness requires the exact filename to pull the file.
PiperOrigin-RevId: 412092347
Fixes the gradle compilation failures.
Gradle dependencies need revising if we want to be using this, as
checkerframework is ahead of their latest version, such that we
can't compile.
PiperOrigin-RevId: 412004021
This change moves methods that are the same in
`TransformerAudioRenderer` and `TransformerVideoRenderer` to
`TransformerBaseRenderer`.
PiperOrigin-RevId: 411758928
Document that apps should retain `GlUtil.Program` while the program is in use,
and keep a reference to attributes/uniforms within the program to make sure
they don't get GC'd causing any allocated buffers passed to GL to become
invalid.
Tested manually by running gldemo and transformer.
PiperOrigin-RevId: 411516894
The features supported by `TranscodingTransformer` are a
superset of those supported by `Transformer` after merging
the video renderers in <unknown commit>. This change removes
`TranscodingTransformer` and adds its features to `Transformer`.
PiperOrigin-RevId: 411072392
Follow-up to a comment on 6f0f7dd1be: Buffers that are useful to pass
to the sample/passthrough pipeline should either contain data or the
end of input flag. Otherwise, passing these buffers along is unnecessary
and may even cause the decoder to allocate a new input buffer which is
wasteful.
PiperOrigin-RevId: 411060709
The `SefSlowMotionVideoSampleTransformer` drops frames by
setting `buffer.data = null` and updates the timestamps of the
frames it keeps. However, these buffers with dropped frames
are still used in the pipeline as they may contain other useful
information, specifically whether the end of the input has been
reached. So, the timestamps should also be updated for the buffers
where frames are dropped.
PiperOrigin-RevId: 410797678
In the near future, we will be able to specify which tests run on this
target explicitly, at which point this ignore will be removed.
PiperOrigin-RevId: 410758785
The `GlUtil` wrappers for attributes/uniforms allocate buffers that are passed into OpenGL, and it seems that the distorted output was caused by these buffers being garbage collected.
The issue was difficult to reproduce manually. Add a test that does repeated transcodes in a loop (marked `@Ignore` for now because it runs for a long time and requires network access so may be flaky) to make it easier to run many transcodes and look for inconsistent output. For now the consistency check is just based on the file size which is very likely to change if the actual video frames change.
Verified that the test fails before, and passes once the attributes and uniforms are stored in fields as in this CL.
PiperOrigin-RevId: 410581706
This change merges `TransformerMuxingVideoRenderer` and
`TransformerTranscodingVideoRenderer` into `TransformerVideoRenderer`.
Besides all features supported by `TransformerTranscodingVideoRenderer`
the new merged `TransformerVideoRenderer` also supports SEF slow motion
flatting without re-encoding like the `TransformerMuxingVideoRenderer`.
To do this, it uses a `SefSlowMotionVideoSampleTransformer` with
the `PassthroughPipeline`.
PiperOrigin-RevId: 410531478
This mime type is technically for the Muxer, and determines
the container used. In the context of the transformer, this can
be thought of more as a container mime type, to avoid confusion
with the video mime type and audio mime type.
Deprecates setOutputMimeType().
PiperOrigin-RevId: 410530707
Simple, initial implementation to allow setResolution()
to set the output height, for downscaling/upscaling.
Per TODOs, follow-up CLs may change layering, add UI,
or allow querying decoders for more resolution options.
PiperOrigin-RevId: 410203343
The decoder writes to `OpenGlFrameEditor`'s input `Surface`
and the `OpenGlFrameEditor` writes to the encoder's input `Surface`.
PiperOrigin-RevId: 409931796
The `VideoSamplePipeline` handles all steps from decoding to
re-encoding that where previously in
`TransformerTranscodingVideoRenderer`. The renderer is now only
responsible for reading the format, reading input, passing it to the
pipeline and passing the pipeline's output to the muxer.
When no transformations are needed, decoding and re-encoding is
skipped using the `PassthroughPipeline`.
PiperOrigin-RevId: 408619407
When dropping the remainder, the decoder and encoder timestamps start diverging after a few buffers when no speed changes are supposed to occur. Tracking the remainder keeps them in sync.
PiperOrigin-RevId: 408341074
* Remove GlUtil.Program String[] constructor to unify and just use the
String constructor.
* Add getAttributeArrayLocationAndEnable() to simplify things a tiny bit.
* Increase usage of constant values.
PiperOrigin-RevId: 407570340
`TransformerAudioRenderer` reads input and passes `DecoderInputBuffer`s to the `AudioSamplePipeline`. The `AudioSamplePipeline` handles all steps from decoding to encoding. `TransformerAudioRenderer` receives `DecoderInputBuffer`s from the `AudioSamplePipeline` and passes their data to the muxer.
`AudioSamplePipeline` implements a new interface `SamplePipeline`. A pass-through pipeline will be added in a future cl.
PiperOrigin-RevId: 407555102
If the number of samples changes, the sizes will help us to verify whether they are just split differently or extra data was added.
PiperOrigin-RevId: 407346280
This helps to prevent issues where decoders can't handle negative
timestamps. In particular it avoids issues when the media accidentally
or intentionally starts with small negative timestamps. But it also
helps to prevent other renderer resets at a later point, for example
if a live stream with a large start offset is enqueued in the playlist.
#minor-release
PiperOrigin-RevId: 406786977
`checkNotNull` should be avoided where possible.
This change adds `@EnsuresNonNull` or `@EnsuresNonNullIf` to configuration methods for fields they initialize.
`checkNotNull` is now avoided for the `@MonotonicNonNull` formats by adding `@RequiresNonNull` annotations.
`checkNotNull` is now avoided for the encoder and decoder in `feedMuxerFromEncoder()`, `feedEncoderFromDecoder()`, `feedDecoderFromInput()`, etc. by creating local variables for `encoder` and `decoder` in `render` after the configuration method calls and passing these as non-null parameters.
PiperOrigin-RevId: 405893824
Also check that the output video MIME type is supported with the given container MIME type in `TranscodingTransformer` and `TransformerBaseRenderer`.
PiperOrigin-RevId: 405645362
When no encoder video MIME type is specified, the `TransformerTranscodingVideoRenderer` now uses the video MIME type of the input for the encoder format.
The input format is now read in a new method `ensureInputFormatRead` which is called before the other configuration methods. This removes the logic for reading the input format from `ensureDecoderConfigured`, because it is now needed for both encoder and decoder configuration but the encoder needs to be configured before GL and GL needs to be configured before the decoder, so the decoder can't read the format.
The width and height are now inferred from the input and the frame rate and bit rate are still hard-coded but set by the `MediaCodecAdapterWrapper` instead of `TranscodingTransformer`.
PiperOrigin-RevId: 405631263
This introduces a new option `setAudioMimeType` in `TranscodingTransformer.Builder` and a corresponding check whether the selected type is supported. This check is done using `supportsSampleMimeType` which is now part of the `Muxer.Factory` and `MuxerWrapper` rather than `Muxer`.
A new field `audioMimeType` is added to `Transformation` and the `TransformerAudioRenderer` uses this instead of the input MIME type if requested.
PiperOrigin-RevId: 405367817
Decoded video frames can be large and there is no need to retrieve the
corresponding ByteBuffer as we render the decoded frames on a surface
for better performance.
PiperOrigin-RevId: 405364950
*** Original commit ***
Migrate callers of ExoPlayer.Builder#build() to buildExoPlayer()
An upcoming change will update build() to return Player.
PiperOrigin-RevId: 401468532
Temporary file copy of Transformer, which uses
TransformerTranscodingVideoRenderer instead of
TransformerMuxingVideoRenderer to transform files. This allows devs to test
transcoding more easily using the Transformer demo, while external engineers
continue to see the completely working Muxing-based transformer.
In the future, this will replace the Transformer class.
PiperOrigin-RevId: 401020893
TracksInfo is very similar to
`MappingTrackSelector.MappedTracksInfo` with some
fields removed to simplify the Player API,
notably it doesn't expose the renderer concept.
A significant difference is the addition of a `selected` boolean
field which avoids having a separate `getCurrentTrackSelection`
API.
This cl is a part of the bigger track selection change,
splitted for ease of review.
In particular, the MediaSession implementation and UI usage
have been slitted in child cls.
Find all cls with the tag:
#player-track-selection
PiperOrigin-RevId: 400937124
