MediaCodecRenderer has already been updated to not rely on the
input stream to mark its samples as decode-only and instead use
a simple time-based comparison to achieve the same effect.
This change makes the same update for all other renderers that
either use the flag directly or forward to a "decoder" instance.
PiperOrigin-RevId: 568232212
If getProgress is blocking whilst the internal thread calls endInternal
(for error or success), the condition is never opened. Related to this,
onCompleted and onError are therefore never surfaced to the app.
progressState is accessed from application and internal threads, so
should be marked volatile to prevent a thread caching the value.
PiperOrigin-RevId: 565720184
When we switch from MUXER_MODE_MUX_PARTIAL_VIDEO to MUXER_MODE_APPEND_VIDEO
`muxedPartialVideo` will already be `true` so `endTrack` method will pass
through this `if(muxedPartialVideo)` check which is incorrect.
PiperOrigin-RevId: 565398117
The logic that handles components' boundaries are grouped together in private
methods, like handling VideoCompositor's output textures.
PiperOrigin-RevId: 565131579
For pause and resume feature we need to use same `MuxerWrapper`
for `remuxing processed video` and then to `process remaining video`.
In order to use same `MuxerWrapper` across `different Exports`
we need to preserve its state.
PiperOrigin-RevId: 564728396
Modifying dumping to not required "released" to be called.
Track index is an arbitrary value based on the order of addTrack calls.
Samples are dumped by track (rather than as soon as they are written),
so it's preferable to use a value that provides more context.
By using the track type as a key, dump files will be more deterministic
and will have more similarities when branched.
PiperOrigin-RevId: 563700982
That said, only SDR is supported for now, so this will always throw if
HDR is input. This will also throw if different ColorInfo values are input
because color SDR mixing (ex. between sRGB and BT709) is not yet supported.
PiperOrigin-RevId: 563457729
For pause and resume feature we will need to build `ExportResult`
from multiple internal Exports. Keeping the `ExportResultBuider` in
`Transformer` class will allow using same builder across different
internal exports.
PiperOrigin-RevId: 563392443
When checking the color transfer, there is no reason that the file should have
exactly two tracks, and this assertion means that this method can't be used
as-is for checking video-only files, for example.
PiperOrigin-RevId: 562813111
Overlays may be overlaid over:
* In VideoFrameProcessor, videos or images (or texture input).
* In Compositor, other videos.
In Compositor, Overlays may consist of video, so it could be confusing
for videoFrameAnchor to contrast with overlayAnchor.
Also, rename overlayAnchor to overlayFrameAnchor, since it's modifying
the anchor in the overlay's frame, so this name seems slightly more precise.
PiperOrigin-RevId: 562004292
Implement VideoCompositor support of:
* Different input and output sizes
* CompositorSettings, to customize output size based on input texture sizes
* OverlaySettings, to place an input frame in an arbitrary position on
the output frame.
Also, refactor Overlay's matrix logic to make it more reusable between
Compositor and Overlays
PiperOrigin-RevId: 561931854
For pause and resume feature we will remux the previously processed
video in the first export and then in the second export we will resume
processing remaining video. For the second export we will have to
set initial timestamp offset so that video samples from the second
export are in continuation to the previously muxed samples.
PiperOrigin-RevId: 561689651
For pause and resume we need to perform multiple intermediate exports.
So moved start() logic into a separate so that it can be reused from
different places with different Compositions and MuxerWrapper.
PiperOrigin-RevId: 561675472
Switch to SparseArray<List> rather than a Map<Integer, List>.
Track indices now determine sample list dumping, reducing flakiness.
PiperOrigin-RevId: 561658586
For pause/resume feature, same MuxerWrapper needs to be
used across intermediate exports. So pass the MuxerWrapper
from Transformer.java
More specifically, when resume() is called
1. Create a MuxerWrapper and remux the previous
video samples (Export 1).
2. User the same MuxerWrapper and start processing
remaining video samples (Export 2).
PiperOrigin-RevId: 561325867
More specifically, this CL
- Defines a VideoGraph interface
- Adds a factory method, to hide the constructors
- Separate out an initialize method that does the real work (following that constructor should not do real work)
- The VideoGraph takes in a list of composition effects. For now, we only use the `Presentation`, if there is one in the list. This means we can take any number of Presentations now, but only the first one will be used.
- Moves the VideoFrameProcessingWrapper to its own file
PiperOrigin-RevId: 561059653
For pause and resume feature we need to find the timestamp of
the last sync sample in an MP4 file. The Mp4ExtractorWrapper
provides an easy to use API on top of Mp4Extractor.
PiperOrigin-RevId: 560113112
The first call to method `registerInputStream` doesn't block.
Later successive calls to the method blocks until the previous register call
finishes.
PiperOrigin-RevId: 559694490
Covers all permutations&combinations of:
* AV asset.
* AV asset with effects.
* AV asset with silence.
* AV asset with silence and effects.
Note that the video is not relevant (therefore transmuxed), but is
needed for silence generation.
PiperOrigin-RevId: 558734593
We will need to build upon original composition
for pause and resume feature and will have to set modified sequences.
In the next CL, will add method to set sequences on builder and will
probably remove it from the constructor parameter (to avoid providing
two APIs for same thing).
PiperOrigin-RevId: 558127334
I wasn't able to reproduce the error in the bug by running videoTranscoding_withTextureInput_completesWithCorrectFrameCountAndDuration
100 times, but I figured this added debug info may help the next time the
issue is seen.
PiperOrigin-RevId: 558118443
In addition to this being how Alpha should be handled, tests do fail under
API 29 without this CL. Unfortunately while these tests do fail under API 29
without this CL, we currently only run this test on API 33 emulators, so we
didn't catch this failure earlier (until compositor tests on mh failed on all
devices under API 29 and succeeded on all at or over 29).
PiperOrigin-RevId: 557781757
Rename AudioMixerImpl to DefaultAudioMixer.
Removes the AudioMixerImpl specific getOutputAudioFormat, as the caller
defines and sets this.
PiperOrigin-RevId: 555887722
To ensure that for each output bitmap from the compostor, the right input
timestamps were used.
Only applied on a subset of tests to avoid needing to upload+maintain
too many files/size in the test binary, especially when it would test
duplicate behavior
PiperOrigin-RevId: 555222530
This means we now require 2+ input frames per input, and compare the primary
stream timestamp with secondary stream timestamps in order to select the
correct output timestamp. We also must release frames and back-pressure as
soon as possible to avoid blocking upstream VFPs.
Also, improve signalling of VFP onReadyToAcceptInputFrame
PiperOrigin-RevId: 553448965
ExoPlayer queues the EOS buffer to the decoder with offset/size/timestamp all
equal to zero, and a EOS flag.
69769c77b3 set TIME_END_OF_SOURCE on the EOS buffer from the extractor.
Queueing the EOS buffer to the decoder with TIME_END_OF_SOURCE causes some
decoders to output wrong timestamps in its output.
This CL replicates what ExoPlayer does in Transformer.
PiperOrigin-RevId: 553104213
This allows for custom implementations of this interface, like
a TestVideoCompositor or partner-implemented implementation
PiperOrigin-RevId: 552541631
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).
This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)
- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
- We bypass the input manager
- The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
- The generated frame is sent to texture output and in turn saved to bitmaps
- We then compare the generated bitmap with the goldens
PiperOrigin-RevId: 551795770
Move shared test logic to the test runner.
This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.
PiperOrigin-RevId: 551536438
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.
PiperOrigin-RevId: 550894935
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.
Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.
Regression tests reproduce failure without fix, but pass with it.
PiperOrigin-RevId: 550853714
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.
PiperOrigin-RevId: 550839156
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.
onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors
PiperOrigin-RevId: 549969933
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.
Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.
PiperOrigin-RevId: 548994584
This serves as a sort of test with more frames for the compositor for now
(before more varied video system tests come later when integrating with
Transformer), showing it doesn't error out and outputs the right
amount of frames.
Due to the VFPTestRunner having a 5s timeout, and mostly due to presubmit
emulators being very slow with OpenGL, there is a sort of limitation
to how many frames this type of test can have, depending on the test target
PiperOrigin-RevId: 548159762
SingleInputVideoGraph implements GraphInput now, so the asset loaders would
interface directly with SIVG, rather than VideoSampleExporter. This is to pave
way for multi-asset video processing.
PiperOrigin-RevId: 547561042
* Allow more than one input bitmap at a time.
* Allow Compositor to take in and set up an Executor. Otherwise,
Compositor resources may be created on one thread and accessed on another.
* Add a Compositor TestRunner to reuse test code more.
* Update VideoFrameProcessingTaskExecutor to use a new onError listener, so
that it's more reusable in non-DVFP contexts, like for Compositor.
PiperOrigin-RevId: 547206053
durationUs is almost always going to be a larger number than the sample
rate, so pass as the main value, rather than the multiplier.
PiperOrigin-RevId: 547193927
TransformerAndroidTestRunner#run will throw any exception that occurs,
so there is no need to assert the result exception is null.
PiperOrigin-RevId: 546923823
Frame buffers are focused, not current. EGL Contexts may be current but this
is a prerequisite for focusing a framebuffer, and this alone isn't
sufficient to make a bitmap.
Clear this up as it can make some complex tests harder to understand, if the
methods needed don't match up in naming.
PiperOrigin-RevId: 546305118
Add documentation for threading requirements at the class level (in
addition to existing documentation on the methods) to improve
discoverablility. Also fix a couple of nits in the javadoc (US English
spelling, avoid passive voice) and in `OnInputFrameProcessedListener`.
PiperOrigin-RevId: 546303732
This allows us to simplify and reduce thread initialization, hops and the
synchronization cost associated with them, especially when more and more VFPs
are used.
PiperOrigin-RevId: 545622169
On a MediaItem change, the input Format (and Effects to apply) may be
different. Therefore the AudioProcessingPipeline must be reconfigured
to determine what processing is active, and what the AudioFormat of the
data output is. In the event that it is different, additional
AudioProcessor instances must be used to ensure the encoder will still
be able to accept the audio buffers.
PiperOrigin-RevId: 544338451
Providing the sync token in the api allows the client to decide which waiting method they would like to use depending on the use case, allowing them to optimise if possible.
PiperOrigin-RevId: 543997311
This fixes a broken javadoc link and makes it possible to customize media
source creation via the asset loader factory without also having to
implementing a decoder factory from scratch.
PiperOrigin-RevId: 543995538
Goal of tests (SequenceExportTest) that use this media is for the
silence and the media to match exactly with audio format, however
`sample_with_increasing_timestamps.mp4` had a different sample rate.
testvid_1022ms.mp4: channel count = 2, sample rate = 44100.
PiperOrigin-RevId: 543458948
MP4 edit lists sometimes ask to start playback between two samples.
If this happens, we currently change the timestamp of the first
sample to zero to trim it (e.g. to display the first frame for a
slightly shorter period of time). However, we can't do this to audio
samples are they have an inherent duration and trimming them this
way is not possible.
#minor-release
PiperOrigin-RevId: 543420218
FallbackListener.onTransformationRequestFinalized() is called from the
AssetLoader thread for audio, and from the GL thread for video.
PiperOrigin-RevId: 542851284
With the upcoming "handle format changes" CL, stereo -> mono audio
would add an AudioProcessor. Robolectric decodes output encoded data,
which crashes some AudioProcessors because the number of frames may not
be an integer.
PiperOrigin-RevId: 542568875
The effects pipeline must receive images in the sRGB colorspace due to the color transfers applied in the shaders. Currently the burden to making sure images are in the right colorspaces falls onto apps. This CL ensures that this is not the case anymore.
PiperOrigin-RevId: 542323613
Audio only tests are now using RAW audio where possible, which is
passed through the Robolectric decoders/encoders, and can be handled by
the AudioProcessor instances accurately.
PiperOrigin-RevId: 541648853
To more accurately describe what they do, especially as Compositor will
starts to use more contexts or threads, and it's important to know what
needs to be reset/recreated/focused before what methods.
PiperOrigin-RevId: 541010135
Issue: When running the Transformer related test cases, the tests are flaky
because the order in which audio and video samples are interleaved seems to
differ in few instances.
Root cause: When running a transformation the sample producer (Asset loader)
and sample consumer (Sample pipeline) both runs on different thread and
theoretically there is no reason for behaviour to be deterministic because
the number of samples produced/written depends on how fast individual thread
works. So it is indeed surprising that test somehow worked deterministically in
majority of instances (may be something to do with Robolectric environment).
Solution: Since we don't expect the order of sample interleaving to be deterministic, make the dumping logic deterministic where all the video
samples will be collected and then dumped together (similarly for audio). This would mean we won't be able to see the interleaving so for that we need to
add separate test case verifying the interleaving logic only.
Pending: Test case for interleaving.
PiperOrigin-RevId: 540930871
The existing NullableType has been deprecated 5 years ago and causes
crashes in Kotlin apps because Kotlin doesn't recognize this annotation
as a nullable type annotation.
While we can't align on a single @Nullable annotation yet, we can at
least replace this one by JSR305's @Nonnull(MAYBE) as it fulfils all
requirements, including full Kotlin compatiblity. To avoid the
cumbersome name, we can redefine it as our own @NullableType
annotation. (We can't use @Nullable to avoid name clashes with the main
@Nullable annotation from AndroidX)
Issue: google/ExoPlayer#6792
PiperOrigin-RevId: 540497469
The test transcodes four media items with distinct effects, keeping one frame
each; extracts the four frames in the produced video, and compares them against
the expected results.
PiperOrigin-RevId: 539697344
This methods currently registers and queues the bitmaps Will need to have a method that just queues bitmaps in future CLs.
#minor-release
PiperOrigin-RevId: 539605628
To ensure no regressions for the potentially confusing pipeline of:
* HDR electrical -> SDR linear EOTF+OOTF, and
* SDR linear -> SDR electrical OETF
PiperOrigin-RevId: 538741079
208eefc0fd introduced using `DefaultDecoderFactory.getDecoderInfo(format) != null` caused certain tests not to be skipped when they were expected to be, creating more mh failures.
PiperOrigin-RevId: 537820370
ExoPlayer extractors (backing `MetadataRetriever`) now parse the color format
from the bitstream so using `MetadataRetriever` should be an equivalent but
more lightweight way to verify the color info.
Also remove try/catch blocks in test code calling into these methods, and add
skipping based on decoder capabilities in the cases where it was missing.
PiperOrigin-RevId: 537789483
In some cases the codec selected for decoding has a different MIME type than
the media. In thoses cases Transformer continued to use the media's MIME type
and that caused codec configuration failures.
Removed `EncoderUtil.findCodecForFormat()` as we stopped using the method it
uses for finding a codec. Plus, the method is only used in the test.
See also `MediaCodecUtil.getALternativeCodecMimeType()`.
PiperOrigin-RevId: 536683663
Tentative/experimental value to reduce codec timeouts. We will reconsider using a larger limit after seeing whether this really does reduce error rate.
PiperOrigin-RevId: 534491615
Mp4Muxer already supports writing Mp4LocationData so added that
as supported Metadata entry.
Support for more Metadata entries will be added in upcoming CLs.
PiperOrigin-RevId: 534473866
Earlier metadata was written multiple times as it came.
With new changes, all the distinct metadata entries will
get collected and will be written at once in the end.
PiperOrigin-RevId: 534088401
This is blocking the upgrade to AGP 8.0.1. The Android Studio Upgrade
Assistant refuses to run with the error:
> Modules have the same package for their `main` and `androidTest`
> artifacts
>
> The package specifications in `AndroidManifest.xml` files define the
> same package for the `main` and `androidTest` artifacts, in the
> following modules: `effect`, `muxer`, `transformer`,
`test_exoplayer_abr`, `test_exoplayer_performance`, and `container`.
This change also adds the `.test` suffix to `test/AndroidManifest.xml`
files where it's missing, because although it's not mentioned in the
Upgrade Assistant error it does cause problems later.
```
> Incorrect package="androidx.media3.database" found in source AndroidManifest.xml: libraries/database/src/test/AndroidManifest.xml.
Setting the namespace via the package attribute in the source AndroidManifest.xml is no longer supported.
Recommendation: remove package="androidx.media3.database" from the source AndroidManifest.xml: libraries/database/src/test/AndroidManifest.xml.
```
Issue: androidx/media#409
PiperOrigin-RevId: 533460180
This allows us to avoid needing a reference to the VideoFrameProcessor, which
can be especially difficult if an App only has a reference to the
VideoFrameProcessor.Factory it passes into Transformer/ExoPlayer.
PiperOrigin-RevId: 533205983
This device failed on HdrEditingTest's exportAndTranscode_hlg10File_whenHdrEditingUnsupported_toneMapsOrThrows
before this CL, and succeeds on that test after this CL.
PiperOrigin-RevId: 532796897
This is a failure only in SSIM, so it seems unlikely we'll prioritize this over
other work or bugs soon. Suppress test failures to reduce triage burden.
PiperOrigin-RevId: 532200729
Allow the VideoFrameProcessor to output multiple textures at a time, so that
lifetime of textures is up to the consumer calling VFP.releaseOutputFrame.
The FinalShaderProgramWrapper also has a new maxCapacity limit added, to ensure
the a reasonable amount of textures is used and avoid using up memory.
PiperOrigin-RevId: 532094256
In the past, the SequenceAssetLoader was released in TransformerInternal
when the export ended.
fc539da061 was made to release the SequenceAssetLoader earlier, when
loading ended. This was causing player release timeouts because the last
AssetLoader in the sequence was released before the SamplePipelines (see
0b40bc37ab for more information).
The code that was releasing the SequenceAssetLoader was first commented
out because we didn't have an understanding of what was happening.
This change removes the early SequenceAssetLoader release all together.
It doesn't have any effect as this code was already commented out.
PiperOrigin-RevId: 532065673
*** Original commit ***
Rollback of 65d5132f76
*** Original commit ***
Create InAppMuxer in transformer
To use the InAppMuxer, the client needs to pass InAppMuxer Factory.
***
***
PiperOrigin-RevId: 531470081
The texture input tests in TransformerEndToEndTest were not passing on
Pixel 7. Implemented a fix and fixed other minor threading issues I
spotted while looking at the code.
PiperOrigin-RevId: 531141659
*** Original commit ***
Create InAppMuxer in transformer
To use the InAppMuxer, the client needs to pass InAppMuxer Factory.
***
PiperOrigin-RevId: 531056436
Tone-mapping an HDR video with MediaCodec on sm-s908u1 was timing out.
The reason for that is that the decoder was dropping frames, and the
ExternalTextureManager was therefore never propagating the end-of-stream
signal.
There was already a workaround for a similar issue but restricted to
sm-f936b. Removed the model check as the bug is probably present on more
devices.
PiperOrigin-RevId: 530639437
Exceptions thrown when switching AssetLoader were not reported through
the error listener. Therefore, the resources were not released and the
export was not ending.
PiperOrigin-RevId: 530626300
If tone-mapping was requested for a device supporting HDR encoding,
isToneMapped was false in VideoSamplePipeline.EncoderWrapper. This was
causing the encoder to expect HDR.
Also did some renamings to improve readability
PiperOrigin-RevId: 530584010
similar error to 9baa6f6be5L `registerInputStream` sets the textureManager, so reordering makes sure the textureManager is set before you set the frame info. This is important for texture input, where the frame info provides the width and height.
PiperOrigin-RevId: 530579584
When we created androidTests, in the past, they always had a URI pointing to a resource, therefore we always had a URI scheme. With texture input, this will not longer be the case (EditedMediaItems's may have URI.EMPTY, which have a null scheme) so we need to check for this so tests don't falsely fail.
PiperOrigin-RevId: 528848411
renderOutputFrame actually renders frames to an output surface. We'll soon have
a releaseOutputFrame method, that would release resources associated with an
output time, so rename this to disambiguate the two methods.
Also rename onOutputFrameAvailable to onOutputFrameAvailableForRendering, to
make it clear this is not available for "release"
This change should be a renaming-only change and have no functional differences.
PiperOrigin-RevId: 527844947
Add `VideoFrameProcessor.registerInputStream()` to signal a new type of input.
And `InputHandler.signalEndOfCurrentInputStream()` to signal to `InputHandler`
partial input stream completion.
Fully processed means after FinalShaderProgramWrapper releases the last frame.
PiperOrigin-RevId: 527356646
In ASwB, all Transformer tests can be run by right-clicking on the project and
clicking "Run all tests". We cannot, however, select only some specific test
files within a project (ex. all non-analysis tests) to run tests on.
Add @Ignore to analysis tests, which are not intended to be run anyways
when determining whether Transformer is working on some device. These
tests also don't have proper skipping logic when a device doesn't support a
format, so they can't be run effectively on low-end devices anyways.
This eases manual testing, for example when debugging whether tests all pass
on a device.
When analysis tests are desired to be run, it should be easy to comment out
the @Ignore.
PiperOrigin-RevId: 527289600
Allow the VideoFrameProcessor to output to a texture without an output surface.
Tested by updating texture output tests to no longer output to a surface.
PiperOrigin-RevId: 527244605
This was only used in EncoderWrapper, so move createSupportedTransformationRequest
from VideoSamplePipeline to EncoderWrapper to reduce its scope.
PiperOrigin-RevId: 527226578
The production code changes are in transformer, but the tests in effect have also been updated to confirm the is no color regression `inputColorInfo.colorTransfer=C.COLOR_TRANSFER_SRGB`
PiperOrigin-RevId: 526950435
Previously, I assumed that SDR contents must be BT709, and HDR contents must be
BT2020. Turns out BT2020 is just wide-gamut, and SDR contents / transfers may be
represented in BT2020 color spaces.
Relax the check, so that we don't throw when valid BT2020 SMPTE 170M contents
are input into effects.
PiperOrigin-RevId: 526668347
Removes glObjectsProvider from `VideoFrameProcessor` and `Effects`. Apps will set the glObjectsProvider on the DefaultVideoFrameProcessor.Factory.Builder when providing a custom DefaultVideoFrameProcessor.Factory, rather than in `Effects`.
PiperOrigin-RevId: 525169059
`findDecoder/EncoderForFormat` seem to be more restrictive than querying
`MediaCodecList` as done for playback, and it's not clear that the cases where
no codec is found correspond to actual cases that would fail given that it
seems this returns an empty string for many cases in production.
Switch to using `MediaCodecUtil` and `EncoderUtil` for querying codecs instead.
PiperOrigin-RevId: 525078321
MediaMuxer does not support out-of-order video frames API 25. Use
a test file with frames in order in
loopingTranscodedAudio_producesExpectedResult. This is necessary for
tests that transmux video samples.
PiperOrigin-RevId: 524309318
A passthrough effect allows for testing having an intermediate
effect injected, which uses different OpenGL shaders from having no
effects.
PiperOrigin-RevId: 524276991
Previously `ChannelMixingAudioProcessor` output float because it was
implemented using the audio mixer's float mixing support.
Move the implementation over to just using the `ChannelMixingMatrix` and make
it publicly visible in the common module so it can be used by apps for both
playback and export.
Also resolve a TODO that no longer had a bug attached by implementing support
for putting multiple mixing matrices to handle different input audio channel
counts, and fix some nits in the test code.
Tested via unit tests and manually configuring a `ChannelMixingAudioProcessor`
in the transformer demo app and playing an audio stream that identifies
channels, and verifying that they are remapped as expected.
PiperOrigin-RevId: 523653901
Simplify the audio encoder input timestamp calculation. The new calculation
avoids drifting by tracking the total number of bytes encoded rather than
tracking the timestamp and remainder separately, and also makes the timestamps
match the decoder output buffer timestamps.
Also switch one of the export tests that was passing through AMR samples over
to using WAVE audio. The problem with using AMR is that the compressed samples
are not necessarily an integer number of audio frames and the shadow decoder
would pass them from input to output, so the audio encoder was receiving
non-integer numbers of audio frames.
Tested by logging the timestamps at the decoder output and encoder input with
forcing transcoding audio, and verifying that after this change the audio
timestamps are no longer off by one.
PiperOrigin-RevId: 523409869
The video asset loader renders decoder output to a surface texture, and if the
video sample pipeline is in the process of updating the surface texture image
at the moment when the asset loader video decoder is released this seems to
cause `MediaCodec.release` to get stuck.
Swap the release order so that we stop updating the texture before trying to
release the codec.
PiperOrigin-RevId: 523401619
Log at debug level immediately when MediaCodec throws. This logging will be
output closer to the time when the error actually happened so should make it
easier to identify the order of components failing.
Downgrade logging of errors after export ends to warning level, as output may
still be fine if there was a problem after exporting completed (though it's
still worth logging a warning as the device may not be in a good state).
PiperOrigin-RevId: 523370457
Before this CL, SurfaceTexture.onFrameAvailable was used to tell whether a frame
was available in the VideoFrameProcessor's output texture. This was incorrect, as
it would rely on having the texture be written to before the
SurfaceTexture.onFrameAvailableListener is invoked, leading to null-pointer-
exceptions on timeouts.
Instead of using DefaultVideoFrameProcessor different interfaces to set that we
want to output to a texture, and get that output texture, use one interface that
sets a listener, and renders to a texture iff that listener is set. As this
listener is executed on the GL thread, this also allows us to no longer need to
expand visibility for the GL task executor and tasks.
PiperOrigin-RevId: 522362101
Handling of the stream offset and start position was unnecessarily
complex and even incorrect. It was going to be an issue for
concatenation of video and image input.
The stream offset is the offset added before decoding/encoding to
make sure it doesn’t fail in case of negative timestamps (which do
rarely occur).
The start position is equal to the stream offset, plus the clipping
start time if the media is clipped.
Before this change:
- Samples were offset by the start position before decoding, and this
offset was removed before muxing.
- The startPosition of the first MediaItem in a sequence was used for
all the MediaItems in this sequence (which is incorrect).
- The stream offset was removed before applying the GL effects and
added back before encoding so that it was not visible to the OpenGL
processing.
After this change:
- The start position is subtracted in the AssetLoader, so that the
downstream components don’t have to deal with the stream offsets and
start positions.
- Decoded samples with negative timestamps are not passed to the
SamplePipelines. The MediaMuxer doesn’t handle negative timestamps
well. If a stream is 10 secondes long and starts at timestamp -2
seconds, the output will only contain the samples corresponding to the
first 8 (10 - 2) seconds. It won’t contain the last 2 seconds of the
stream. It seems acceptable to remove the first 2 seconds instead.
PiperOrigin-RevId: 520916464
Releasing the player once a sequence has ended seems to make our
emulator tests flaky. Comment out until we find the cause. The player
will still be released from TransformerInternal, when the export ends.
PiperOrigin-RevId: 520886181
*** Original commit ***
Rollback of fc539da061
*** Original commit ***
Release SequenceAssetLoader when ended
Before, the SequenceAssetLoader was released at the end of the export.
Release resources earlier if possible.
***
***
PiperOrigin-RevId: 520036433
*** Original commit ***
Release SequenceAssetLoader when ended
Before, the SequenceAssetLoader was released at the end of the export.
Release resources earlier if possible.
***
PiperOrigin-RevId: 519993003
Previously, we always used ImageReader to read from the output of DefaultVideoFrameProcessor, for pixel tests. This has a limitation of not being
able to read HDR contents, so that we couldn't support HDR pixel tests.
Reading from a texture allows us to use glReadPixels to read from
DefaultVideoFrameProcessor, and build upon this to implement HDR pixel tests. We do
still want tests for surface output though, because real use-cases only will output
to Surfaces.
Also, add some tests for outputting to textures, since this test infrastructure is
a bit complex.
PiperOrigin-RevId: 519786535
The failure reason (no network) will be logged in the test output, but the test will not be marked as failing, rather skipped.
PiperOrigin-RevId: 518262203
This was trimmed using the Transformer Demo, from ~4 seconds to 300ms
Otherwise, this test file will be quite large, and folks who clone the repo will
have to clone this content as well. Trimming the file also allows the test to
take less time, while still testing the core purpose of the file, to confirm that 4k extraction/decode/GL/encode/muxing works.
PiperOrigin-RevId: 517408463
This test file used to be remote, which led to various errors in loading the file or timing out during the muxer or overall transformer.
export4k60 test timing on Pixel 7:
|Condition |elapsedTimeMs|AS Test "Duration" (s)|
|--------------------------|-------------|----------------------|
|local file (this CL) |4253 |14 |
|Remote file, googleguest |5510 |17 |
|Remote file, 4G |11423 |25 |
PiperOrigin-RevId: 516815462
- To support looping EditedMediaItemSequences, we need a way to tell the
AssetLoader that a sample couldn't be consumed and that it should retry
later. This is necessary in case we don't know yet whether the looping
sequence should load more samples because the other sequences haven't
made sufficient progress yet.
- The decision on whether to consume a sample is based on its timestamp
so it needs to be available.
PiperOrigin-RevId: 516546026
This test is not run on emulator, triage or postsubmit, and does not
provide value to the library in it's current state. It could be
deleted, however it still may have use for analysis of exported files
on MH.
PiperOrigin-RevId: 516491160
Split test/ TransformerEndToEndTest into SingleMediaItemEndToEndTest and
SingleSequenceEndToEndTest to reduce the file size and split the tests
by category.
PiperOrigin-RevId: 515039502
Callbacks onTrackCount and onTrackAdded can be called simultaneously
from different threads.
Before this fix, it was possible for the MuxerWrapper and
FallbackListener track count to never be set, or to be set
with incorrect values.
PiperOrigin-RevId: 514779719
- This is to make sure we know about all the tracks before initializing
the SamplePipelines. This allows to set the muxer and the fallback
listener track count before the SamplePipelines are built.
- As a result, the test files had to be updated because the order in
which the tracks are written has changed.
- The ImageAssetLoader also had to be updated to call onOutputFormat
repeatedly until it returns a non-null SampleConsumer.
- Also fix the trackCount sent to the muxer and fallback listener. The
correct track count can be computed now that we know about all the
tracks before building the SamplePipelines.
PiperOrigin-RevId: 514426123
All audio tracks should either all be transcoded or all be transmuxed.
Same for video tracks.
To achieve this, simplify the behaviour of transmuxAudio/Video.
PiperOrigin-RevId: 513809287
If the Metadata passed to SegmentSpeedProvider is null, then the
SegmentSpeedProvider will always return 1f from getSpeed.
Initializing a SpeedChangingAudioProcessor requires a SpeedProvider.
Once configured,this audioProcessor is always active, so buffers are
passed through it. Because getSpeed is always 1, the processor performs
a no-op, but still has to do a buffer copy for each buffer.
By not initializing the audio processor when metadata is null, this
copy can be skipped and the audio pipeline is more performant.
Note: This change does not affect the multiple media-item case, which
is not supported with speed changes, as per Transformer API
documentation.
PiperOrigin-RevId: 513261811
We shouldn't have this logging unless we really need it to debug
a specific problem, as it can be noisy (even at debug level).
PiperOrigin-RevId: 512904412
Uses the first mediaItem's format as the output format.
If there is `Presentation` supplied in the `Composition.effects`, add it as the
last effect of the first EditedMediaItem.
PiperOrigin-RevId: 512082659
Add format codec info, which can make test skipping checks more similar to the
actual Transformer decoder checks.
Also for the test file, the actual format was 720p, but somehow the file name and
media metadata indicated 1080p. This format mismatch led to some decoding errors,
so fix the format (and associated errors). This also allows us to remove the
exception catch in ForceInterpretHdrVideoAsSdrTest, which was included due to
errors from the incorrect format.
PiperOrigin-RevId: 511809507
- Split the transmux setting into transmuxAudio and transmuxVideo. This
is more flexible for apps and will also be useful for unit testing
(particularly as we can't test video transcoding on Robolectric at the
moment).
- Move these settings to Composition. It makes sense for these settings
to be next to forceAudioTrack. Apps may also want to set these
settings based on the current Composition's MediaItems.
- Add a Composition.Builder because Composition now contains a few
optional fields.
PiperOrigin-RevId: 511708618
Logcat had the following lines, with no other information.
```
DefaultEncoderFactory: Encoders removed for resolution:
DefaultEncoderFactory: Encoders removed for bitrate:
DefaultEncoderFactory: Encoders removed for bitrate mode:
```
PiperOrigin-RevId: 511470231
Implement getMediaFormatInteger, a helper method simulating mediaformat.getInteger(name, defaultValue).
This reduces the API 29 restriction from MediaFormatUtil.getColorInfo to API 24, in
particular removing the method-based restriction to a constant-based restriction,
so that we can reduce usage of the API 29 class.
This also allows us to slightly simplify prior use-cases where we'd check
containsKey and getInteger to have a default value.
PiperOrigin-RevId: 511184301
When clipping a MediaItem with start time > 0, the audio was ending
before the video. This is because:
- Audio timestamps are computed based on the sample sizes, with a start
time set to streamOffsetUs (i.e. the streamStartPositionUs is not
taken into account).
- The SamplePipeline was subtracting streamStartPositionUs from the
timestamps before sending the samples to the muxer.
- As a result, the audio timestamps were shifted by
streamStartPositionUs, while they should be shifter by streamOffsetUs.
PiperOrigin-RevId: 511175923
* Account for `AssetLoader` output types.
* Consider cases that are not audio/video specific.
* Use `Format#sampleMimeType` for track specific conditions to check.
* Untangle `SamplePipeline` initilization from `AssetLoader` state.
PiperOrigin-RevId: 511020865
- Add silent audio when the output contains an audio track but the
current MediaItem doesn't have any audio.
- Add an audio track when generateSilentAudio is set to true.
PiperOrigin-RevId: 511005887
Previously, this was limited to API 29. Expand this to all API versions.
Also, update the test to:
(1) skip based on SDR format input instead of HDR format input
(2) Check the exception message in order to disambiguate between the decoder tone
mapping error, and general video format support error.
PiperOrigin-RevId: 511002218
Unstuck the muxer if the next timestamp in the track with the minimum
timestamp is larger than this minimum timestamp plus
MAX_TRACK_WRITE_AHEAD_US.
PiperOrigin-RevId: 510977088
Before, if the upstream AssetLoader provides HDR to the VideoSamplePipeline when
HDR_MODE_EXPERIMENTAL_FORCE_INTERPRET_HDR_AS_SDR is requested, the
VideoSamplePipeline would attempt to tell the AssetLoader to output SDR, which
could be accomplished via MediaCodec tone-mapping in the AssetLoader.
However, this makes an assumption of the AssetLoader implementation, and
AssetLoaders may not all implement support for decoder tone-mapping. Remove javadoc
attempting to explain how AssetLoaders (ex. custom ones) could behave.
PiperOrigin-RevId: 510956820
This method uses sampleRate, channelCount and pcmEncoding, so passing
AudioFormat is easier.
This will lead into a future change that builds the
encoderInputAudioFormat from encoder.getConfigurationFormat()
PiperOrigin-RevId: 510956177
Rename ScaleToFitTransformation to ScaleAndRotateTransformation.
This better represents the operations that can be accomplished using this
effect. The name was originally named ScaleToFit* because it's not obvious how
to scale to fit using OpenGL, and this effect handled the scaling to fit in a way that no other MatrixTransformations did.
However, it's hard to discover how to rotate when skimming names of effects, so
it's probably more useful to convey that this effect rotates, than that it
scales to fit.
PiperOrigin-RevId: 510480078
Format.toString unfortunately doesn't log colorInfo, and as Format holds a very
large set of values, it's unclear that it should. ColorInfo is useful for codec
exceptions though, so log this in ExportException.createForCodec.
PiperOrigin-RevId: 510475520
Otherwise, a lack of HDR decoding support will result in the tests checking output files for HDR output, like HdrEditingTest.transform_noRequestedTranscode_hdr10File_transformsOrThrows, failing.
PiperOrigin-RevId: 510213020
This allows us to fix usage of HDR_MODE_EXPERIMENTAL_FORCE_INTERPRET_HDR_AS_SDR.
Before, this was checked in the VideoSamplePipeline, which no longer decides on the decoder configuration input format.
PiperOrigin-RevId: 510142097
Some of the test methods in `TransformationTest` don't check
capabilities. Add a check just scoped to decoding (not checking
encoding, but the default encoder factory may fall back).
PiperOrigin-RevId: 510088562
* Moved the logic to SamplePipeline.
* Pass the requested values via Format.
* Moved exception throwing inside the methods.
* Build up the mimeTypesToCheck as a set - removing possible duplicate
checks.
* Simplified logic that calls the findSupportedMimeType method.
* Improved javadoc.
PiperOrigin-RevId: 509594062
It may be confusing for 3P apps, for us to have separate error codes for (1) if
encoding HDR is not supported at all by the device, and (2) if encoding the
format, which happens to be HDR, is not supported by the device. Instead, we can
communicate this in the error message.
PiperOrigin-RevId: 509188666
- For single-asset, the behavior stays the same. Transcode if and only
if it's necessary,
- For constrained multi-asset, always transcode, except if the setter to
transmux is set. This is to avoid failing if a MediaItem that doesn't
require transcoding is followed by a MediaItem that does require
transcoding.
PiperOrigin-RevId: 508097798
It doesn't actually make sense for them to be placed in the Transformer, because the error's root causes are actually only in codecs. Also, a few codec errors were
repeated, so deduplicate these instances
PiperOrigin-RevId: 506937695
Based on [this conversation thread](https://chat.google.com/room/AAAA--f88ao/76Rem_cRCK8), I've opted to update the existing FrameProcessor.create() rather than deprecate it, as it is unlikely to be in use by apps outside google3.
PiperOrigin-RevId: 506920930
* Overload added `(cause, errorCode, isVideo, isDecoder, details)`,
where `details` is a string of values to be added to the error message
of the `TransformationException`.
* Overload with `MediaFormat` and `mediaCodecName` moved to
`DefaultCodec`, because all usages of that overload were from
`DefaultCodec`, and this allows a simplified API because of internally
stored values.
* `mediaCodecName` removed from overload that takes a `Format`.
* Reordered `createForCodec` parameters.
PiperOrigin-RevId: 506895268
In encoding small and odd-numbered resolutions, like `316x61` ([this image](https://upload.wikimedia.org/wikipedia/commons/6/65/100winners.png)), the current fallback logic prefers a software encoder to hardware ones. The assumption was, the encoder factory applies the encoder size alignment and changes the resolution to `316x60` for SW encoders and `316x64` for HW ones. SW encoders is selected because the supported resolution 60 is closer to requested 61, than the hardware supported 64.
This change changes the default encoder selection process to only expose hardware encoders if there is any.
PiperOrigin-RevId: 506879983
With multi-asset, the sample pipelines can process more than one
MediaItem. The renaming makes it clear that the format passed to the
SamplePipeline constructors is the one corresponding to the first
MediaItem. Indeed, the first format is the one used to configure the
SamplePipelines.
PiperOrigin-RevId: 506879260
The prior version (with the call to createEncodingException) could
never occur as select...SupportedMimeType already checks for HDR
editing support. This change ensures we throw before creating an
encoder, gives a better error code and allows future simplifications
around createForCodec (see child CL).
PiperOrigin-RevId: 506308290
Transformer callbacks will take a Composition instead of a MediaItem.
Apps should be able to see what this Composition contains.
PiperOrigin-RevId: 505976561
Add checks to GL tone-mapping pixel tests, to ensure the device's decoder, API
version, and OpenGL implementation support GL tone-mapping before attempting it.
These tests should be run on mobile harness, to detect per-device failures, and
so are moved to transforemr/mh. Per b/263395272, these tests should ultimately
be in an effect/mh directory.
PiperOrigin-RevId: 505749974
ParcelFileDescriptor is quite specific and removing it allows to have
less overloads of startTransformation.
A follow-up CL will undeprecate the overload that takes a MediaItem.
PiperOrigin-RevId: 505670321
Also add a Builder to EditedMediaItem to avoid having a constructor with
many optional parameters, or a chain of constructors.
PiperOrigin-RevId: 504588544
Transformer convention has been to avoid splitting into packages unless
necessary.
See https://jlbp.dev/JLBP-2: "Prefer fewer packages over more packages
to avoid unnecessarily publicizing internal details, since any
dependency across package boundaries needs to be public.
PiperOrigin-RevId: 503463294
This is an AssetLoader that wraps a sequence of AssetLoaders. It will
be used for constrained multi-asset.
This class can currently only concatenate media items with the exact
same format.
PiperOrigin-RevId: 502525796
Also remove usages of TransformationRequest convenience methods
(setScale, setRotationDegrees and setResolution).
Some usages of setResolution can't be removed yet because they are used
for fallback.
PiperOrigin-RevId: 502415748
This is needed for constrained multi-asset to shift the timestamps of
the media items that are not the first in the sequence.
PiperOrigin-RevId: 502409923
This class may be removed soon, but in the meantime clarify this method's
javadoc summary fragment to make it marginally clearer that this only affects
the "displayed" height instead of the "encoded" height.
PiperOrigin-RevId: 501582219
Small refactoring change to simplify the logic a tiny bit by removing one
unnecessary variable. No functional change intended.
PiperOrigin-RevId: 501566533
It makes more sense to have it in the AssetLoader now that decoding is
done there, particularly as the decoder factory should rarely be
customized.
Also remove the setter from the AssetLoader.Factory because it's
uncommon to have setters in a factory.
PiperOrigin-RevId: 501244974
This brings together the multiple details about a muxer track, and
reduces the need for additional variables and more complicated track
tracking.
PiperOrigin-RevId: 499872145
The old names are not really correct anymore because:
- The Audio/VideoTranscodingSamplePipelines do not decode anymore.
- The pipelines now mux the encoded data.
PiperOrigin-RevId: 499498446
Whilst testing fallback functionality, I found that we were
aggressively reducing the resolution if it was not supported. A quick
test found that we could reduce by a much smaller increments.
Performance wise it appears these checks are incredibly quick.
The code for checking supported sizes was duplicated, with one case
having a bug because of this duplication (2/3 case). This CL abstracts
this into a loop.
PiperOrigin-RevId: 499497646
When a listener method is deprecated, the new method should (by
default) called through to the deprecated one. This is because any
class that implements the method that is now deprecated needs to still
receive that callback.
It appears when onTransformationError(MediaItem, Exception) was
deprecated in favour of onTransformationError(MediaItem,
TransformationException), this deprecation was the wrong way round, and
the newer callback - onTransformationError(MediaItem,
TransformationResult, TransformationException) continued this mistake.
This CL now corrects this.
PiperOrigin-RevId: 498221504
Events on the wrapper should be propagated to TransformerInternal as
soon as they occur, switching round the process so TransformerInternal
does not have to query MuxerWrapper.
This CL is a prerequisite for the child CL, where MuxerWrapper can
simplify the internal state and logic.
PiperOrigin-RevId: 497267202
Before, it was possible for onDurationUs() and onAllTracksRegistered()
to be called before onTrackAdded() because they are called from
different threads. onDurationUs() and onAllTracksRegistered() are called
from the Transformer internal thread, and onTrackAdded() is called from
the playback thread.
PiperOrigin-RevId: 497102556
AssetLoader declares the tracks with a setTrackCount() method. Setting
the track count on the MuxerWrapper is easier than calling
registerTrack() as many times as the number of tracks.
PiperOrigin-RevId: 496933501
AssetLoader declares the tracks with a setTrackCount() method. Setting
the track count on the FallbackListener is easier than calling
registerTrack() as many times as the number of tracks.
PiperOrigin-RevId: 496919969
The TransformationRequest passed to FallbackListener (and
createSupportedTransformationRequest) can have null for values that
are inferred from source. Within fallback, this can be height, width,
video mime type and audio mime type (HDR mode is not linked to source).
requestedFormat has these values populated from source, and
supportedFormat then finds the closest supported values to the
requested.
If any of the values in supportedFormat do not match the
requestedFormat, then this method would build upon the
TransformationRequest and update ALL possible fallback fields. This is
a problem because the fallback listener compares the original request
to the fallback one and notifies about all the fields that have
changed.
This CL changes this so that only the values that are not the same as
requested are changed in the supported request that is given to the
fallback listener.
PiperOrigin-RevId: 496908492
When trying to run the test on Android Studio, error "incompatible
types: Buffer cannot be converted to ByteBuffer" is logged. This is
because ByteBuffer.flip() returns a Buffer (and not a ByteBuffer).
Annotation @CovariantReturnType on ByteBuffer.flip() should resolve this
automatically but it doesn't seem supported at the moment.
PiperOrigin-RevId: 496894723
Note that we simply use GlEffectsFrameProcessor in-app / GL tone-mapping, so PQ->SDR tone-mapping isn't yet implemented.
Tested manually using the demo on Pixel 7, to confirm that device and in-app tone
mapping behave similarly.
PiperOrigin-RevId: 496700231
Adds the AudioMixerAlgorithm interface which allows for specialized
implementations of audio mixing that also efficiently convert between
source and mixing formats.
Initial implementation has two algorithms:
1. Float -> float (with channel mixing)
2. S16 -> float (with channel mixing)
PiperOrigin-RevId: 496686805
To do that, rename PROGRESS_STATE_NO_TRANSFORMATION to
PROGRESS_STATE_NOT_STARTED and update Javadoc of ProgressState to not be
Transformer specific.
PiperOrigin-RevId: 496653460
This should allow us to focus on HDR failures instead of network buffering
failures when debugging HDR issues.
These files are each used on several files, so it should be more worth the
test binary impact to move these files to local first.
Locally, tests did take less time after this diff
PiperOrigin-RevId: 496398130
The MediaSourceFactory won't be used by the other AssetLoaders
In order to do that, ExoPlayerAssetLoader has been made public, and the
DefaultAssetLoaderFactory has become a wrapper around
ExoPlayerAssetLoader.
PiperOrigin-RevId: 496386853
Otherwise, the decoders are not captured. It works at the moment for the
video decoder because decoding is still done on the sample pipeline but
it will moved to the AssetLoader soon.
PiperOrigin-RevId: 495275575
`AudioProcessor`s expect direct buffers. This shouldn't make any functional difference in our code, but a custom audio processor might try to access the buffer from JNI in which case a direct byte buffer is more efficient.
PiperOrigin-RevId: 495241669
`SilentAudioGenerator` could output a fractional audio frame, and this could cause downstream components to throw because of trying to read a complete audio frame but only seeing a partial one.
Calculate the output buffer size based on the frame size (which is a no-op for stereo 16-bit audio) and calculate a total number of frames to output then multiple by the frame size.
PiperOrigin-RevId: 494992941
The duration in TransformerInternal.ComponentListener is set on the
Transformer internal thread, and is read on the playback thread. Making
this field volatile ensures that the playback thread reads the updated
value.
PiperOrigin-RevId: 493908385
Rename ERROR_CODE_OUTPUT_FORMAT_UNSUPPORTED to
ERROR_CODE_ENCODING_FORMAT_UNSUPPORTED.
This makes the error code more consistent with ERROR_CODE_DECODING_FAILED on the
decoding side. Also, the error code is in the "Encoding errors (4xxx)" section,
so muxer errors probably should be in the "Muxer errors (7xxx)" section instead.
Additionally, no muxer errors currently seem to use
ERROR_CODE_OUTPUT_FORMAT_UNSUPPORTED, so this should be a safe change.
PiperOrigin-RevId: 493322880
This is necessary to move video slow motion flattening to the
AssetLoader because this step can change the duration. As we use the
duration before flattening to calculate the progress, we must also use
the position before flattening.
PiperOrigin-RevId: 493291990
Modify FrameProcessor and MatrixTextureProcessor interfaces to support
different input and output color transfers. Does not implement conversion between
color ranges (ex. HDR and SDR), but should allow for conversion between color
transfers of the same color range (ex. HLG and PQ).
This supports in-app tone mapping, where we need a single FrameProcessor to
input HDR color transfers (ex. HLG/PQ) and output SDR (ex. gamma2.2). This also
supports previewing, where we need a single FrameProcessor to be able to input HLG
and output PQ.
Manually tested by confirming colors still look right on SDR and HDR videos
with a rotation and color affect applied.
PiperOrigin-RevId: 493108678
From this CL on, FrameProcessor listeners will be invoked from an Executor that
is passed in when creating the FrameProcessor.
GlTextureProcessor needs to invoke the ErrorListener on the said Executor too.
PiperOrigin-RevId: 493018583
This is necessary to move video decoding to the AssetLoader. Otherwise,
if the decoder max pending frame count is reached, the AssetLoader will
stop queuing frames to the pipeline, and process data will not be called
anymore.
PiperOrigin-RevId: 492392621
EncoderUtil using a static list to cache encoders means it's not possible to
use a different set of encoders for different tests when running all
robolectric tests together.
PiperOrigin-RevId: 491959350
Bypass Z Fold 4 HDR10 tone-mapping bug by limiting the max frame count to 12.
This passed with a value of 14, and failed with a value of 15, but I figured I'd use 12 just to be safe.
PiperOrigin-RevId: 491684058
Main change:
- Removed `Codec.EncoderFactory.createForVideoEncoding`'s argument of a list
of allowed MIME types
- Moved the check for whether a video MIME type is supported to VTSP
PiperOrigin-RevId: 491611799
This thread just starts the player and handles the player callbacks for
now. Sample pipelines are still run on the playback thread.
PiperOrigin-RevId: 491299671
This reduces the priority to best effort (from the default that seems to be best effort), and allows us to run SSIM even on 8k24fps video. Without this CL,
start()'ing a second codec may result in a MediaCodec.CodecException.
Tested to confirm that transformation8k24():
* fails deterministically without this CL, or with KEY_PRIORITY set to 0.
* succeeds deterministically after this CL (~18s on Samsung Z Fold 4)
PiperOrigin-RevId: 490570416
We cannot check this in code, due to DEVICE_INITIAL_SDK_INT being a @SystemApi, and
reflection being a bit risky/unstable.
PiperOrigin-RevId: 490537916
The bug has since been fixed.
The values still could change, as the API is labelled as @UnstableApi, so it's
probably fine to leave the <p> tag mostly as is.
PiperOrigin-RevId: 490509205
This allows us to release both codecs used in SSIM when one fails to
configure() or start().
Tested and confirmed that on Samsung Galaxy Z Flip 4, running all
TransformationTest.java tests, tests after transform8k24() fails to start the
2nd codec:
* Before this CL, all fail.
* After this CL, all pass.
PiperOrigin-RevId: 490461560
This exception is a bit shorter and more clear (and is more clear that this is a
test issue, as opposed to the prior issue that was thrown as an ExoPlayer
DataSourceException, which may seem like a legitimate Transformer failure)
PiperOrigin-RevId: 490252772
Following naming conventions throughout AndroidTestUtil, REMOTE files should have REMOTE instead of ASSET. Update the URI and FORMAT names accordingly.
PiperOrigin-RevId: 490237673
AssetLoader will have multiple implementations and be customizable. We
want to remove the responsibility of computing the progress from this
class and centralize the logic in TransformerInternal.
PiperOrigin-RevId: 488608890
Not setting the color info results in a missing "colr" box in the produced
container, under file/moov/trak/mdia/minf/stbl/stsd/hvc1. This means extractors
will not be able to find out the transcoded file is HDR.
In `Transformer`, this means it can't transcode this transcoded file, because
it currently relies on the container bearing HDR info to construct the
transcoding sample pipeline.
PiperOrigin-RevId: 487276712
In startTransformation method we were throwing UnsupportedEncodingException (IOException) when mediaItem with unsupported arguments is passed.
Changed this to IllegalArgumentException which seems more logical here.
PiperOrigin-RevId: 487259296
This logic is currently in the player renderers. With multi-asset, the
renderers will go into the AssetLoader, which shouldn't be responsible
for muxing.
PiperOrigin-RevId: 486860502
Problem: We are initialising muxer as soon as we start the transformation. Now the startTransformation() method can be called from main thread, but muxer creation is an I/O operation and should be not be done on main thread.
Solution: Added lazy initialisation of muxer object. The actual transformation happens on background thread so the muxer will be initialised lazily from background thread only.
Another way was to provide an initialize() method on MuxerWrapper which will explicitly initialise muxer object but with this approach the caller need to call the initialise method before calling anything else. With current implementation the renderers are calling MuxerWrapper methods on various callbacks (Not sequentially) and also we are sharing same muxer with multiple renderers so It might become confusing for the caller on when to call the initialise() method. Also there are few methods on MuxerWrapper which dont really need muxer object. So in short it might make MuxerWrapper APIs more confusing.
Validation: Verified the transformation from demo app.
PiperOrigin-RevId: 486735787
This should be necessary to ensure decoders see fewer errors.
Setting this resulted in removing native_dequeueOutputBuffer errors on OMX.MTK decoders for in-app tone mapping prototyping.
PiperOrigin-RevId: 486715941
Although it can be useful to check the output format, it's not required or needed.
For some AudioProcessor implementations, it is stated/obvious that
the output format will match the input, in which case there is no
a need to check the return value.
#cleanup
PiperOrigin-RevId: 483403679
The reason for making the Muxer public is that we want to add an option
to disable or configure the timer that will throw when the muxer doesn't
receive any data for a given period of time.
PiperOrigin-RevId: 482199360
- The naming DefaultMuxer is more consistent with the rest of
Transformer codebase (e.g. DefaultEncoderFactory).
- By hiding the implementation details of DefaultMuxer, the transition
to in-app Muxer will be seamless for apps using DefaultMuxer.
- The current plan is that DefaultMuxer will become the in-app muxer.
PiperOrigin-RevId: 481838790
Assert that tone mapping is applied when an HDR edit cannot be HDR, but is successfully tone mapped. Meanwhile, assert that fallback, which is applied after codec configuration (which throws the "Tone-mapping requested but not supported by the decoder" error) is not applied when that error is called.
PiperOrigin-RevId: 478762951
Rename test files to avoid substrings that can be implied by the directory name,
like "Transformation" and "Test"
No functional changes. Renaming-only.
PiperOrigin-RevId: 477724724
* Transform the intermediate color space to linear SDR by applying the SMPTE 170M EOTF and OETF.
* Use linear colors for the color filter pixel tests and update all golden bitmaps.
PiperOrigin-RevId: 476124592
*** Original commit ***
Handle int instead of byte in SSIM.
The value of pixels are converted to integers at the point of use,
move this logic to the initialisation step.
This is a prerequisite step for testing SSIM calculation, which
will lead on to some SSIM improvements being verifiable.
Tested manually and SSIM values match for the same video
before and after this change.
***
PiperOrigin-RevId: 473259446
The value of pixels are converted to integers at the point of use,
move this logic to the initialisation step.
This is a prerequisite step for testing SSIM calculation, which
will lead on to some SSIM improvements being verifiable.
Tested manually and SSIM values match for the same video
before and after this change.
PiperOrigin-RevId: 473231779
Test that HDR editing succeeds on devices supporting HDR editing, tone maps on
devices supporting tone mapping, and throws exceptions on all other devices.
Also, only restrict HDR editing and tone mapping support to API 31+ only when
transcoding, not for all transformations.
PiperOrigin-RevId: 472958965
shouldPassthrough's internal checks seem to be check whether we should *not*
pass through, which seemed a bit like a confusing double-negative to me.
shouldTranscode is slightly more clear, by instead returning true when we do
want to transcode.
No functional changes intended.
PiperOrigin-RevId: 471753771
3b0d2c1586 made `shouldPassthrough` always return false for `enableHdrVideoEditing`:
>We force using `FrameEditor` (no passthrough) to avoid the need to select another edit operation, and use the new shaders. The `EGLContext` and `EGLSurface` also need to be set up differently for this path.
However, this was introduced before the `videoNeedsEncoding` setting was introduced in 3f615040c0. That setting should apply to HDR videos as much as SDR videos.
PiperOrigin-RevId: 471569853
Adds a method to FrameProcessor.Listener to be called when an
output frame is available and a method releaseOutputFrame in
FrameProcessor allowing the caller to trigger release of the
oldest available output frame at a given timestamp. Late frames
or frames with unset release times are dropped in the
FinalMatrixTransformationProcessorWrapper.
More than one output frame can become available before they are
released if the penultimate GlTextureProcessor is capable of producing
multiple output frames. Processing continues while waiting for
releaseOutputFrame to be called. Frame release tasks are prioritized
over other tasks.
PiperOrigin-RevId: 468473072
createSupportedTransformationRequest is more accurate than
createFallbackTransformationRequest, as a TransformationRequest will be returned
regardless of whether any fallback is applied.
PiperOrigin-RevId: 466641277
Previously, this feature interpreted SDR signals as HDR when called. Now, only HDR
streams are interpreted as HDR, so the javadoc should be updated. Not yet removing
this method, as there are still some loose ends to finish up (ex. PQ support, e2e
tests).
PiperOrigin-RevId: 466425738
While HDR is most closely tied to the color transfer (ex.
COLOR_TRANSFER_SDR is the only one explicitly mentioning dynamic
range), technically color spaces may be associated with HDR as well,
like BT.2020 commonly being used for HDR rather than BT.709 for SDR.
Therefore, it's more specific to mention just that the transfer is HDR.
PiperOrigin-RevId: 466316960
This allows the GlEffectsFrameProcessor to later handle HLG and PQ
differently, or limited and full color range differently.
No functional change intended in this CL.
PiperOrigin-RevId: 466070764
In the case where this check fails, the downstream frame processor chain won't be able to handle the incoming (SDR) data anyway as we've already set it up for HDR.
PiperOrigin-RevId: 465584814
`requestCalculateSsim` more clearly represents the intention of the caller.
Also rephrase the javadoc to simplify it and make it more precise.
PiperOrigin-RevId: 465575578
