This is an internal refactor with no logic changed.
There is a duplication in information and a lack of consistency around
the use of test assets in transformer androidTest. This change
refactors each asset to be defined as its own AssetInfo, with the other
relevant parts stored alongside the uri.
Once this is in place, we can consider other useful functionality, such
as having boolean flags for what tracks an asset has, helper methods
for whether an asset is local or remote, and more. This will reduce the
manual overhead necessary to use more assets in tests, and in
particular leads towards easily using new & existing assets in
parameterized tests.
PiperOrigin-RevId: 644040595
This CL is a pure refactor of the existing tests, iterative changes
will be done in follow-ups.
The only logic change is having all MP4 assets have the presentaton of
height=360. The old code had some with height 480.
PiperOrigin-RevId: 643020773
- Video release should check for buffer timestamp (which is renderer-offsetted), rather than the frame timestamp
- ImageRenderer should report ended after all of it's outputs are released, rather than when finished consuming its input.
Add tests for timestamp handling
PiperOrigin-RevId: 642587290
Before this change, the timestamps output from composition playback is offset
with the renderer offset. After this change, the offset is removed and the
timestamp behaviour converges with Transformer, that is, the timestamps of
video/images frames will follow that of the composition. For example, with a
composition of two 10-s items, clipping the first with 2s at the start, the
timestamp of the first frame in the second item, will be 8s.
PiperOrigin-RevId: 641121358
Add SeparableConvolution.configure(inputSize) to allow effect configuration
depending on input dimensions.
Add LanczosResample.scaleToFit method to scale input images to fit inside
given dimensions.
PiperOrigin-RevId: 640498008
Switch from 4-channel RGBA_16F lookup texture to 1-channel R_16F.
Do not use a bitmap when creating the lookup table texture.
Instead, fill the texture directly.
Do not manually convert 32-bit float to 16-bit. Instead, let OpenGL
libraries do this for us.
PiperOrigin-RevId: 639717235
Degammaing has been removed in cb4b2ea55c. The goldens for
TransformerHdrTest (previously TransformerSequenceEffectTestWithHdr)
were not regenerated because the test wasn't running due to its name
(fixed in e41a966237).
PiperOrigin-RevId: 638645635
Add DefaultVideosFrameProcessor experimental flag that controls
whether input Bitmaps are sampled once for a repeating sequence of
output frames with the same contents, or once for each output frame.
PiperOrigin-RevId: 637921350
Only sample from input bitmap when the input image has changed.
Introduce GainmapShaderProgram.newImmutableBitmap API that signals
input bitmap changes to GainmapShaderProgram (DefaultShaderProgram).
PiperOrigin-RevId: 637920207
PQ and HLG have different luminance ranges (max 10k nits and max 1k nits resp). In GL, colors work in a normalised 0 to 1 scale, so for PQ content, 1=10k nits and and for HLG content, 1=1k nits.
This cl scales and normalises PQ content appropriately so that all HDR content works in the HLG luminance range. This fixes two things
1. Conversions between HLG and PQ are "fixed" (before the output colors looked too bright or too dark depending on which way you are converting)
2. color-altering effects will be able to work consistently across HLG and PQ content
1 is tested in this cl. 2 will be tested when ultra HDR overlays are implemented, both cases have been manually tested to ensure the output looks correct on a screen.
PiperOrigin-RevId: 636851701
Remove redundant test logic to add file size to ExportResult because
the file size is already added to export result as part of an export
finishing.
PiperOrigin-RevId: 636499236
Fragment shaders in OpenGL ES shader language aren't guaranteed
to support highp, required to correctly represent pixel coordinates
inside large images (e.g. 1920x1080).
This change moves coordinate mirroring for images out of fragment shader.
Fixes http://Issue: androidx/media#1331
PiperOrigin-RevId: 635732208
If MediaCodec allocates passes an image buffer with a cropped region,
SurfaceTexture.getTransformMatrix will cut off 2 pixels from each dimensions.
The resulting videos will appear a little stretched.
This patch inspects the SurfaceTexture transform matrix, and guesses what the
unscaled transform matrix should be.
Behind experimentalAdjustSurfaceTextureTransformationMatrix flag
PiperOrigin-RevId: 635721267
For each item, AudioGraphInput now pads the input audio track with silence
to the duration given in onMediaItemChanged.
Possibly resolves Issue: androidx/media#921 .
PiperOrigin-RevId: 634753721
This class is not ready for production app usage yet, so it is still
marked `@RestrictTo(LIBRARY_GROUP)` for now. Apps can experiment with it
in a non-prod context by suppressing the associated lint error.
* Issue: androidx/media#1014
* Issue: androidx/media#1185
* Issue: androidx/media#816
PiperOrigin-RevId: 633921353
The process crashes unexpectedly on these devices.
The new changes skips running these tests instead of marking them pass, when
the device needs to be skipped.
PiperOrigin-RevId: 633183638
Two devices were producing B-frames earlier and were causing
frame mismatch. So we had added a workaround for them.
Those devices does not produce B-frames now
after disabling high profile on them, so we don't need workaround now.
PiperOrigin-RevId: 633127755
Use different textures in calls to queueInputTexture(). Allows the texture to be deleted one it is used in transformer and effect.
PiperOrigin-RevId: 632430866
After this CL, DVFP waits for flushing until all frames registered previously
arrives.
Previously, ETM records the difference between the number of registered frames,
and the number of frames arrivd on the SurfaceTexture, when flushing. (Note
that ETM is flushed the last in the chain, as flushing is done backwards from
FinalShaderProgramWrapper). ETM then waits until the number of frames arrive
after flush.
The normal flow is, MediaCodecVideoRenderer (MCVR) registers a new decoded
frame, in `processOutputBuffer()` to DVFP, MCVR call `codec.releaseOutputBuffer()`
to have MediaCodec render the frame, and then the frame arrives in DVFP's ETM.
However there might be a discrepancy. When registering the frame, ETM records
the frame on the calling thread, ~instantly. Later when the rendered frame
arrive, ETM records a frame is available on the task executor thread (or
commonly known as the GL thread). More specifically, when a frame arrives
in `onFrameAvailableListener`, ETM posts all subsequent processing to
the task executor. When seeking, the task executor is flushed as the first
step. It might be a frame that has already arrived on ETM, and the processing
of such frame has already been queued into the task executor; only to be
flushed as a result of flushing the task executor. If this happens, the frame
is considered to be never have arrived. This causes a freeze on the app,
because ETM'll wait until this frame arrives to declare flushing has completed.
PiperOrigin-RevId: 631524332
The second stage of the changes remove the conversion to linear colors in the SDR effects pipeline by default.
also resolves Issue: androidx/media#1050
PiperOrigin-RevId: 630108296
Also adds first frame rendered test for playing back compositions.
- This test checks the output pixels using an `ImageReader` to retrieve the
output bitmap
PiperOrigin-RevId: 630100817
Part of a two stage change to remove the conversion to linear colors in the SDR effects pipeline by default. Changes the boolean to an intdef, introducing a third option that gets all sdr input into the same colorspace.
This is a planned API breaking change, but this change should not change the behavior of the pipeline.
PiperOrigin-RevId: 629013747
also makes the setter more flexible by ignoring the value of the setter when the output is hdr rather than throwing (since all HDR content must be have a linear color space)
PiperOrigin-RevId: 627388436
Instead of initializing the video sink outside the renderer with an
empty format for composition preview, we initialize it in the renderer
with the input format for video.
PiperOrigin-RevId: 627313708
the output format conatins key information about the output of the assetloader being hdr, so we must signal the output format, not the input format to the sample exporter
fixes mh ultraHdr test failures.
Also discovered images created are very device specfic so got rid of the pixel tests, we have pixel tests in the effects library that cover the same case.
PiperOrigin-RevId: 619899249
This property is transformer specific and does not belong to
muxer interface.
This is to eventually replace muxer interface in Transformer module with
muxer interface in Muxer module.
PiperOrigin-RevId: 618895836
This is the most widely-used test-skipping method I'm aware of, so I figured this
would be a great method to scale usage of AssumptionViolatedException.
PiperOrigin-RevId: 618160931
This also makes it more consistent with newer testId tests that don't have nullness, so tests don't look inconsistent, and was pretty easy to do :P
PiperOrigin-RevId: 616856888
This was broken by 2fa38d15dd, which added an assumes
method, but missed the `!` operator for the tests where HDR is
unsupported. Add an assumes method that assumes a lack of support
for HDR.
Tested on the failing device (Pixel 7) and confirmed this fixes
the test on that device, to throw AssumptionViolatedException
instead of allowing test logic to run after failing the
assumption.
PiperOrigin-RevId: 616846588
Plumbing hdrMode through the default asset loader factory via the constructor is problematic because it breaks API boundaries. It means there is another way to set hdrMode outside of Composition.java and TransformationRequest.java, which is error prone and cause problems if someone an app starts customizing the assetloaderfactory. It also means custom asset loaders can't receive this information without hacking around.
The introduction of the composition-level settings class makes this approach easily extensible for other settings applied on the composition level but use in an individual asset level basis (e.g. ultraHDR support).
PiperOrigin-RevId: 611466920
Add abilitiy to use real surfaces in instrumentation tests
using the ActivityScenarioRule and an activity class for testing
purposes.
PiperOrigin-RevId: 611421490
This new test is for `ExoPlayer.setVideoEffects()`. It plays the
one-second-long video, applies an overlay that prints the video frame timestamp
onto the frame, captures the output frame and compares the captured output
frame with golden.
PiperOrigin-RevId: 610781590
MediaCodec docs already allude to potentially mismatching H.264 level
between container and bitstream. Relax the initialization data check to
reflect this.
PiperOrigin-RevId: 608942322
Our previous test video was difficult to use for testing our tone-mapping
algorithm, because it didn't have many different colors. Use a better
video for tone-map tests, by having one with more different colors
PiperOrigin-RevId: 606274843
TestUtil class is more appropriate for the given method.
In the next CL, the only method in FileUtil.java will be moved back
into transformer library and the FileUtil class will be removed.
PiperOrigin-RevId: 605648034
Before supporting transmuxing when both no op effects and regular rotations are set, move setting the muxerWrapper rotation out of shouldTranscodeVideo() to ensure the muxerWrapper rotation is only set at the appropriate times.
This cl also ensures the state between the muxerWrapper and the list of video effects is consistent by clearing the list of videoEffects in trim optimization. If trim optimisation is being applied, then EditedMediItem.effects.videoEffects only contains no-op effects or regular rotations that get be applied in the muxer wrapper. Therefore, we should clear the list of video effects to ensure that no effect gets applied twice.
PiperOrigin-RevId: 604292052
The `MediaMuxer` (FrameworkMuxer) supports AV1 mime type from API 34.
For this to work track `Format/MediaFormat` must have `CSD`
data.
Change also include providing AV1 option in demo app.
Transmuxing of an AV1 mime type input will not work because
`Extractor` does not extract AV1 CSD data.
Verified changes manually via demo app.
PiperOrigin-RevId: 603002380
FileUtil.java can be extended to add more methods for validating
output file. This will be used in Transformer's robolectric tests and
instrumentation tests hence need to move to a common module.
PiperOrigin-RevId: 602789503
Previously, input assets had to be all SDR or all HDR.
After this CL, if tone-mapping is requested, HDR and SDR may mix in any order. If tone-mapping is not requested, SDR may precede HDR, but not vice versa, until SDR to HDR tone-mapping is implemented
Some changes to accomplish this include:
1. Inputting the decoded format's color to VideoFrameProcessor.registerInputStream
for each stream.
2. Calculating the estimated decoded format's color for each stream, by estimating
it based on MediaCodec tone-mapping.
PiperOrigin-RevId: 602747837
This allows us to also plumb whether we're doing MediaCodec tone-mapping,
which will be used in a follow-up CL in the ExoAssetLoaderVideoRenderer.
PiperOrigin-RevId: 601774435
Allow setAdditionalRotationDegrees to be called with same rotation after tracks added. This is needed for processes that mux files partially trim optimization so they don't error out after hitting the check state
Manually tested to ensure trim optimization succeeds, automated test added here as well
PiperOrigin-RevId: 601081778
This device doesn't seem to be capable of simultaneous encode/decode at this
resolution. We don't have a good way to check the capability (we are already
checking separate decode/encode capability) so just skip this test to save time
triaging its failures.
PiperOrigin-RevId: 600399564
This file is CBR encoded with LAME, so it has an `Info` header (the CBR
equivalent to `Xing`).
A follow-up change will use this file in `Mp3ExtractorTest`.
Issue: androidx/media#878
PiperOrigin-RevId: 595938327
Mp4Muxer does not support out of order B-frames. Currently it
silently writes out of order B-frames, producing an invalid file (with
negative sample durations).
Although `Mp4Extractor` is somehow able to process this invalid file and
`Exoplayer` is able to play it but that is unexpected.
The `sample.mp4` test file contains B frames. Other test files does not
contain `H264 video + AAC audio` format hence created a new test file by
running `sample.mp4` via `Transformer` after applying some effects.
PiperOrigin-RevId: 594016144
Instead, for input videos, use the colorInfo provided by the extractor. Similarly, for input images, use sRGB, the only color currently in use.
Textures do still need the input ColorInfo provided though.
PiperOrigin-RevId: 592875967
Checking the output format's mime type may skip tests more often than we'd like,
because we may desire using a lower-spec output mimetype than what's passed in, if
based on the input's HDR mimetype value.
Therefore, update this output format to null, for tone-mapping tests
PiperOrigin-RevId: 592855713
Despite GL 3.0 not being required on API 29+, it is experimentally
determined to always be supported on our testing devices, on API 29+.
That said, still fall back to OpenGL 2.0 if 3.0 is not supported,
just in case.
PiperOrigin-RevId: 590569772
Added a new ABANDONED option so one can tell the difference between when the optimization has been requested but not applied vs not requested at all. also changed the ordering do better represent the hierarchy of failure modes
PiperOrigin-RevId: 588720513
This helped me debug which bitmap was actually failing. Otherwise, you need to clear all bitmaps on the device relating to this test, then adb pull all bitmaps and see the last uploaded one to see, which is much more confusing.
PiperOrigin-RevId: 588374081
The issue that motivated adding this (frames unexpectedly being dropped by the
decoder) has been addressed, so we can turn off the logging to reduce
unnecessary allocations during transformation. We can easily turn on debug
logging in future as needed by setting `DebugTraceUtil.DEBUG = true`.
Also avoid allocations for string building at logging call sites by passing a
format string for extra info. Formatting the string now only happens when
debugging is turned on.
Tested manually by running transformations in the new state (DEBUG = false) and
with debugging turned on.
PiperOrigin-RevId: 585666349
The decoder and encoder won't accept high values for frame rate, so avoid
setting the key when configuring the decoder, and set a default value for the
encoder (where the key is required).
Also skip SSIM calculation for 4k, where the device lacks concurrent decoding
support.
PiperOrigin-RevId: 585604976
Composition and EditedMediaItemSequence don't allow empty lists in their main
constructors, so shouldn't the vararg API. This is more inline with Effective
Java item 53.
PiperOrigin-RevId: 583415124
Skip wearable devices, which don't have the same CDD requirements on codecs from Android 5 (API 21).
Limit an existing skipping case to pre API 33 devices, so that we can catch failures on Android 13 (API 33) onwards, and add OnePlus 9 Pro which also times out calculating SSIM. Remove TODOs for removing test/SSIM skipping now they are restricted to API version.
PiperOrigin-RevId: 581985554
This device over-reports encoding capabilities, so skip the test.
This is a Pixel device that was upgraded to Android 12 (API 31/32) so the issue should only affect devices that didn't take OTAs.
PiperOrigin-RevId: 581981073
Changes includes:
1. Add resume flow.
2. Change demo app code to resume export.
3. Changes in test infra to trigger resume.
4. E2E Test cases
PiperOrigin-RevId: 579895744
