the output format conatins key information about the output of the assetloader being hdr, so we must signal the output format, not the input format to the sample exporter
fixes mh ultraHdr test failures.
Also discovered images created are very device specfic so got rid of the pixel tests, we have pixel tests in the effects library that cover the same case.
PiperOrigin-RevId: 619899249
If the mixer is reset without resetting maxPositionOfRemovedSources and
then reused, the value of maxPositionOfRemovedSources can be outdated,
leading to an incorrect number of bytes being output by the mixer.
PiperOrigin-RevId: 619832502
This CL aims to separate Fragmented MP4 related logic in a separate public class.
Earlier all the logic was in a single class `Mp4Muxer`.
PiperOrigin-RevId: 619206661
This property is transformer specific and does not belong to
muxer interface.
This is to eventually replace muxer interface in Transformer module with
muxer interface in Muxer module.
PiperOrigin-RevId: 618895836
The immediate plan is to replace `Transformer Muxer interface`
with this one.
It was not straight forward to move/change `Transformer Muxer interface`
hence this intermediate step.
The follow up CL will also remove fragmented MP4 related code from
`Mp4Muxer` and move it into `FragmentedMp4Muxer implements Muxer`.
PiperOrigin-RevId: 618789265
This is the most widely-used test-skipping method I'm aware of, so I figured this
would be a great method to scale usage of AssumptionViolatedException.
PiperOrigin-RevId: 618160931
Issue: androidx/media#725
Ideally the test to transmux a `ts` file (having 3 bytes NAL start code)
should go into `Mp4MuxerEndToEndTest`but `Mp4MuxerEndToEndTest` uses `MediaExtractor` and `MediaExtractor` is returning samples with `4 byte NAL start code` which will not exercise the newly added code path.
Hence the test is added in `TransformerWithInAppMuxerEndToEndTest` which
internally uses `media3 extractor` and feeds samples with `3 bytes NAL start code`
only.
PiperOrigin-RevId: 617985866
Seeking will consist of the following steps:
- Block the AudioGraph input data.
- Flush the AudioGraph.
- Seek the ExoPlayers.
- Unblock the AudioGraph input data.
PiperOrigin-RevId: 617868124
This also makes it more consistent with newer testId tests that don't have nullness, so tests don't look inconsistent, and was pretty easy to do :P
PiperOrigin-RevId: 616856888
This was broken by 2fa38d15dd, which added an assumes
method, but missed the `!` operator for the tests where HDR is
unsupported. Add an assumes method that assumes a lack of support
for HDR.
Tested on the failing device (Pixel 7) and confirmed this fixes
the test on that device, to throw AssumptionViolatedException
instead of allowing test logic to run after failing the
assumption.
PiperOrigin-RevId: 616846588
When flushing the AudioGraph, the AudioMixer will be preserved but the
sources will be recreated. This is probably a bit less efficient but
makes the logic simpler. Indeed, if we had to keep the sources alive,
we would need to add a way to reconfigure them with a new timestamp for
seeking. We would also need to change the way sources are ended because
they are currently removed when they are ended. Also, it is acceptable
to have a small delay when seeking, which means that performance is less
critical than for playback.
PiperOrigin-RevId: 615775501
getSupportedEncoderNamesForHdrEditing became getSupportedEncodersForHdrEditing at some point, but this comment wasn't updated before...
PiperOrigin-RevId: 615712707
Pass firstAssetLoaderOutputFormat to videoSampleExporter for non-video use cases, so that the downstream components like the videoFrameProcessor can be set up with the right output color. Surface creation is still in the VSP so can't do this for all use cases currently.
also moves getDecoderOutputColor() to TransformerUtil, since it is used in multiple places and doesn't make sense for once to have reference to the other.
PiperOrigin-RevId: 613113958
Plumbing hdrMode through the default asset loader factory via the constructor is problematic because it breaks API boundaries. It means there is another way to set hdrMode outside of Composition.java and TransformationRequest.java, which is error prone and cause problems if someone an app starts customizing the assetloaderfactory. It also means custom asset loaders can't receive this information without hacking around.
The introduction of the composition-level settings class makes this approach easily extensible for other settings applied on the composition level but use in an individual asset level basis (e.g. ultraHDR support).
PiperOrigin-RevId: 611466920
Add abilitiy to use real surfaces in instrumentation tests
using the ActivityScenarioRule and an activity class for testing
purposes.
PiperOrigin-RevId: 611421490
This new test is for `ExoPlayer.setVideoEffects()`. It plays the
one-second-long video, applies an overlay that prints the video frame timestamp
onto the frame, captures the output frame and compares the captured output
frame with golden.
PiperOrigin-RevId: 610781590
Previously, the track format was used in VideoSampleExporter. Now, we use a
simulated decoder output format.
As the last expected change for this bug, also adds release notes
PiperOrigin-RevId: 609080629
MediaCodec docs already allude to potentially mismatching H.264 level
between container and bitstream. Relax the initialization data check to
reflect this.
PiperOrigin-RevId: 608942322
Our previous test video was difficult to use for testing our tone-mapping
algorithm, because it didn't have many different colors. Use a better
video for tone-map tests, by having one with more different colors
PiperOrigin-RevId: 606274843
TestUtil class is more appropriate for the given method.
In the next CL, the only method in FileUtil.java will be moved back
into transformer library and the FileUtil class will be removed.
PiperOrigin-RevId: 605648034
The new API will take both `creation time` and `modification time`.
Till now, Mp4Muxer wrote `modification time` in both
`creation time` and `modification time` field, which was
incorrect.
PiperOrigin-RevId: 605590623
Change the file extension retrieval back to how it was before 5488d33da8 to reduce the change of false negatives in `isImage()`
PiperOrigin-RevId: 605281186
Earlier implementation compared the whole file against a golden
data. The new implementation compares only the metadata being tested.
This will avoid updating the golden data when any unrelated change
(unrelated to scenario being tested) is made.
Added a separate test to compare the whole output file against a golden data.
PiperOrigin-RevId: 604692985
Before supporting transmuxing when both no op effects and regular rotations are set, move setting the muxerWrapper rotation out of shouldTranscodeVideo() to ensure the muxerWrapper rotation is only set at the appropriate times.
This cl also ensures the state between the muxerWrapper and the list of video effects is consistent by clearing the list of videoEffects in trim optimization. If trim optimisation is being applied, then EditedMediItem.effects.videoEffects only contains no-op effects or regular rotations that get be applied in the muxer wrapper. Therefore, we should clear the list of video effects to ensure that no effect gets applied twice.
PiperOrigin-RevId: 604292052
Transformer export and ExoPlayer previewing both read inputColorInfo from
registerInputStream now, instead of maintaining a consistent input color
throughout multiple streams in a sequence.
Therefore, we can remove inputColor-related arguments and methods now.
PiperOrigin-RevId: 603423509
The `MediaMuxer` (FrameworkMuxer) supports AV1 mime type from API 34.
For this to work track `Format/MediaFormat` must have `CSD`
data.
Change also include providing AV1 option in demo app.
Transmuxing of an AV1 mime type input will not work because
`Extractor` does not extract AV1 CSD data.
Verified changes manually via demo app.
PiperOrigin-RevId: 603002380
FileUtil.java can be extended to add more methods for validating
output file. This will be used in Transformer's robolectric tests and
instrumentation tests hence need to move to a common module.
PiperOrigin-RevId: 602789503
Previously, input assets had to be all SDR or all HDR.
After this CL, if tone-mapping is requested, HDR and SDR may mix in any order. If tone-mapping is not requested, SDR may precede HDR, but not vice versa, until SDR to HDR tone-mapping is implemented
Some changes to accomplish this include:
1. Inputting the decoded format's color to VideoFrameProcessor.registerInputStream
for each stream.
2. Calculating the estimated decoded format's color for each stream, by estimating
it based on MediaCodec tone-mapping.
PiperOrigin-RevId: 602747837
* getPcmFormat Util method already sets the mime type as AUDIO_RAW.
* static final AudioFormat consts improve test case readability.
PiperOrigin-RevId: 602740007
If trim optimisation is being applied, then `EditedMediItem.effects.videoEffects` only contains no-op effects or regular rotations that get be applied in the muxer wrapper. Therefore, we should clear the list of video effects to ensure that no effect gets applied twice.
This fixes requested rotations being applied twice in trim optimization.
Manually tested to ensure all combinations of trimming+muting+rotating works at intended.
PiperOrigin-RevId: 601853203
Many usages are needed to support other deprecations and some
can be replaced by the recommended direct alternative.
Also replace links to deprecated/redirected dev site
PiperOrigin-RevId: 601795998
since we now can support taking in the inputColor upon registering the stream, there is no need to hardcode the image input color anymore. We should now be able to support switching between texture and image input which we couldn't before, but this is untested and not necessary.
PiperOrigin-RevId: 601784149
This allows us to also plumb whether we're doing MediaCodec tone-mapping,
which will be used in a follow-up CL in the ExoAssetLoaderVideoRenderer.
PiperOrigin-RevId: 601774435
Allow setAdditionalRotationDegrees to be called with same rotation after tracks added. This is needed for processes that mux files partially trim optimization so they don't error out after hitting the check state
Manually tested to ensure trim optimization succeeds, automated test added here as well
PiperOrigin-RevId: 601081778
[Android best
practices](https://developer.android.com/media/optimize/sharing#android_8_81_and_9)
recommend disabling B-frames on API 27, but some devices output B-frames anyway
when H.264/AVC High profile is selected. Add a workaround to force these
devices not to use high profile, to prevent B-frames being output.
`MediaMuxer` likely handles B-frames on these specific devices, but this change
allows the migration to default to in-app muxing to take place without
introducing errors, and it's a temporary workaround until B-frames are properly
supported in the in-app muxer.
PiperOrigin-RevId: 600422238
This device doesn't seem to be capable of simultaneous encode/decode at this
resolution. We don't have a good way to check the capability (we are already
checking separate decode/encode capability) so just skip this test to save time
triaging its failures.
PiperOrigin-RevId: 600399564
Although not public, documenting what happens in each state allows for
better code understanding at a glance.
As part of this, refactored the #getProgress method to highlight that
the other options are "non standard".
PiperOrigin-RevId: 599505369
Moves setting bitmapFactory options from BitmapLoader to DatasourceBitmapLoader
BitmapLoader is a general interface for bitmap loading that could use loading implementations other that BitmapFactory, with the rise of Glide being a loader of choice. It's best to correct this interface so that it remains generic
We can't deprecate easily because the other loadBitmap method in that case has a default implementation that relies on the first one, so the change is still breaking. BitmapLoader is public api in common, but it's @UnstableAPI and hasn't been around for very long (be38670391 added it), so it seems this is the best way forward.
PiperOrigin-RevId: 597897098
This file is CBR encoded with LAME, so it has an `Info` header (the CBR
equivalent to `Xing`).
A follow-up change will use this file in `Mp3ExtractorTest`.
Issue: androidx/media#878
PiperOrigin-RevId: 595938327
Mp4Muxer does not support out of order B-frames. Currently it
silently writes out of order B-frames, producing an invalid file (with
negative sample durations).
Although `Mp4Extractor` is somehow able to process this invalid file and
`Exoplayer` is able to play it but that is unexpected.
The `sample.mp4` test file contains B frames. Other test files does not
contain `H264 video + AAC audio` format hence created a new test file by
running `sample.mp4` via `Transformer` after applying some effects.
PiperOrigin-RevId: 594016144
Instead, for input videos, use the colorInfo provided by the extractor. Similarly, for input images, use sRGB, the only color currently in use.
Textures do still need the input ColorInfo provided though.
PiperOrigin-RevId: 592875967
Replace the event for notifying fallback to cover codec initialization in
general (but keeping the list of errors).
Add a flag to control whether to try non-primary codecs, with the same
documentation as the similar flag in ExoPlayer's renderer.
Make the class final as it shouldn't be necessary to subclass it.
PiperOrigin-RevId: 592869868
Checking the output format's mime type may skip tests more often than we'd like,
because we may desire using a lower-spec output mimetype than what's passed in, if
based on the input's HDR mimetype value.
Therefore, update this output format to null, for tone-mapping tests
PiperOrigin-RevId: 592855713
Move `inputColorInfo` from `VideoFrameProcessor`'s `Factory.create` to `FrameInfo`,
input via `registerInputStream`.
Also, manually tested on exoplayer demo that setVideoEffects still works.
PiperOrigin-RevId: 591273545
Despite GL 3.0 not being required on API 29+, it is experimentally
determined to always be supported on our testing devices, on API 29+.
That said, still fall back to OpenGL 2.0 if 3.0 is not supported,
just in case.
PiperOrigin-RevId: 590569772
Also adds an alternate way to configure the AudioGraph.
Apps should no longer need to ensure that inputs have the same sample
rate.
PiperOrigin-RevId: 588747431
Added a new ABANDONED option so one can tell the difference between when the optimization has been requested but not applied vs not requested at all. also changed the ordering do better represent the hierarchy of failure modes
PiperOrigin-RevId: 588720513
This helped me debug which bitmap was actually failing. Otherwise, you need to clear all bitmaps on the device relating to this test, then adb pull all bitmaps and see the last uploaded one to see, which is much more confusing.
PiperOrigin-RevId: 588374081
* @Before and @After usage. [1].
* static fields before non-static. [2].
@Before method should typically be paired with an @After method,
focused on ensuring that the component is released regardless of what
the test does.
In tests, inlining final class variables is preferrable [1]. In general
things like the file path should be part of the test (the @Rule means
we don't need before/after) if only used once.
Statically importing values and using them directly is preferable to
having a variable declared as non-final that's effectively final,
because from a readability perspective someone can see (the caps) that
the value is final static and immutable, so doesn't have to check if
it's changed/reassigned.
PiperOrigin-RevId: 586697887
The issue that motivated adding this (frames unexpectedly being dropped by the
decoder) has been addressed, so we can turn off the logging to reduce
unnecessary allocations during transformation. We can easily turn on debug
logging in future as needed by setting `DebugTraceUtil.DEBUG = true`.
Also avoid allocations for string building at logging call sites by passing a
format string for extra info. Formatting the string now only happens when
debugging is turned on.
Tested manually by running transformations in the new state (DEBUG = false) and
with debugging turned on.
PiperOrigin-RevId: 585666349
The decoder and encoder won't accept high values for frame rate, so avoid
setting the key when configuring the decoder, and set a default value for the
encoder (where the key is required).
Also skip SSIM calculation for 4k, where the device lacks concurrent decoding
support.
PiperOrigin-RevId: 585604976
Using `Integer.MAX_VALUE` risks causing arithmetic overflow in the codec
implementation.
Issue: androidx/media#810
#minor-release
PiperOrigin-RevId: 585104621
These usages have no need for the double ended input functionality. All
other usages across media3 are ConcurrentLinkedQueue.
PiperOrigin-RevId: 584841104
Composition and EditedMediaItemSequence don't allow empty lists in their main
constructors, so shouldn't the vararg API. This is more inline with Effective
Java item 53.
PiperOrigin-RevId: 583415124
When transmuxing, the `EncodedSampleExporter` maintains a queue of input
buffers that get filled with encoded data by the asset loader. The number of
buffers was limited to avoid using more and more memory if producer (asset
loader) gets far ahead of the consumer (exporter).
Previously this limit was fixed at 10 buffers, but increasing the number of
buffers can make some transmux operations much faster. Allow allocating between
a min and max number of buffers, and also set a target allocation size beyond
which new buffers can't be allocated. This allows audio formats which require
many small buffers to be processed more quickly, while preventing allocating
too much memory for hypothetical very high bitrate formats.
'Remove video' edits on local videos in particular get much faster, because
audio buffers are very short and there are lots of them. With a sample 10
minute video, a 'remove video' edit took 2 seconds (36 seconds before this
change). With a sample 1 minute removing video took 0.25 seconds after this
change (2.5 seconds before).
The speed improvement is smaller for other types of edits that retain the video
track. Transmuxing a 10 minute video retaining the video track took 26 seconds
(40 seconds before).
PiperOrigin-RevId: 583390284
MIME types are case-insensitive, but none of the many existing
comparisons across our code base take this into account. The
code can be made more robust by normalizing all MIME types at the
moment they are first set into a class/builder and adding toLowerCase
as part of the normalization.
Most concretely, this fixes an issue with playing HLS streams via
the IMA SDK where the stream MIME type is indicated with all lower
case "application/x-mpegurl", which failed the MIME type comparison
in DefaultMediaSourceFactory.
PiperOrigin-RevId: 582317261
Skip wearable devices, which don't have the same CDD requirements on codecs from Android 5 (API 21).
Limit an existing skipping case to pre API 33 devices, so that we can catch failures on Android 13 (API 33) onwards, and add OnePlus 9 Pro which also times out calculating SSIM. Remove TODOs for removing test/SSIM skipping now they are restricted to API version.
PiperOrigin-RevId: 581985554
This device over-reports encoding capabilities, so skip the test.
This is a Pixel device that was upgraded to Android 12 (API 31/32) so the issue should only affect devices that didn't take OTAs.
PiperOrigin-RevId: 581981073
The dump file diff as part of this change is because using AudioGraph
means the 2nd item is automatically edited to match the AudioFormat of
the 1st item {44.1KHz mono}, rather than {48KHz stereo}.
Manually verified that for the 2nd item, data output:
* Before: 66936 bytes (16734 frames) output = 348_625us of audio.
* After: 30750 bytes (15375 frames) output = 348_639us of audio.
The small final buffer is caused by SonicAudioProcessor outputting all
pending data when EOS queued, and is WAI.
PiperOrigin-RevId: 580494578
Changes includes:
1. Add resume flow.
2. Change demo app code to resume export.
3. Changes in test infra to trigger resume.
4. E2E Test cases
PiperOrigin-RevId: 579895744
When transmuxing, we usually only need to offset the timestamp by the position of a mediaItem in a sequence.
Trim optimization introduces another type of offset: for the transmux of the second part of the video we need to offset the timestamps by the total duration already trancoded by transformer.
PiperOrigin-RevId: 576134656
Move remote 8K file to local and trim to 320ms.
Trim done with ffmpeg:
`ffmpeg -i {remote_file} -t 0.3 -c:v copy -c:a copy 8k24fps_300ms.mp4`
PiperOrigin-RevId: 569449962
MediaCodecRenderer has already been updated to not rely on the
input stream to mark its samples as decode-only and instead use
a simple time-based comparison to achieve the same effect.
This change makes the same update for all other renderers that
either use the flag directly or forward to a "decoder" instance.
PiperOrigin-RevId: 568232212
If getProgress is blocking whilst the internal thread calls endInternal
(for error or success), the condition is never opened. Related to this,
onCompleted and onError are therefore never surfaced to the app.
progressState is accessed from application and internal threads, so
should be marked volatile to prevent a thread caching the value.
PiperOrigin-RevId: 565720184
When we switch from MUXER_MODE_MUX_PARTIAL_VIDEO to MUXER_MODE_APPEND_VIDEO
`muxedPartialVideo` will already be `true` so `endTrack` method will pass
through this `if(muxedPartialVideo)` check which is incorrect.
PiperOrigin-RevId: 565398117
The logic that handles components' boundaries are grouped together in private
methods, like handling VideoCompositor's output textures.
PiperOrigin-RevId: 565131579
For pause and resume feature we need to use same `MuxerWrapper`
for `remuxing processed video` and then to `process remaining video`.
In order to use same `MuxerWrapper` across `different Exports`
we need to preserve its state.
PiperOrigin-RevId: 564728396
Modifying dumping to not required "released" to be called.
Track index is an arbitrary value based on the order of addTrack calls.
Samples are dumped by track (rather than as soon as they are written),
so it's preferable to use a value that provides more context.
By using the track type as a key, dump files will be more deterministic
and will have more similarities when branched.
PiperOrigin-RevId: 563700982
That said, only SDR is supported for now, so this will always throw if
HDR is input. This will also throw if different ColorInfo values are input
because color SDR mixing (ex. between sRGB and BT709) is not yet supported.
PiperOrigin-RevId: 563457729
For pause and resume feature we will need to build `ExportResult`
from multiple internal Exports. Keeping the `ExportResultBuider` in
`Transformer` class will allow using same builder across different
internal exports.
PiperOrigin-RevId: 563392443
When checking the color transfer, there is no reason that the file should have
exactly two tracks, and this assertion means that this method can't be used
as-is for checking video-only files, for example.
PiperOrigin-RevId: 562813111
Overlays may be overlaid over:
* In VideoFrameProcessor, videos or images (or texture input).
* In Compositor, other videos.
In Compositor, Overlays may consist of video, so it could be confusing
for videoFrameAnchor to contrast with overlayAnchor.
Also, rename overlayAnchor to overlayFrameAnchor, since it's modifying
the anchor in the overlay's frame, so this name seems slightly more precise.
PiperOrigin-RevId: 562004292
Implement VideoCompositor support of:
* Different input and output sizes
* CompositorSettings, to customize output size based on input texture sizes
* OverlaySettings, to place an input frame in an arbitrary position on
the output frame.
Also, refactor Overlay's matrix logic to make it more reusable between
Compositor and Overlays
PiperOrigin-RevId: 561931854
For pause and resume feature we will remux the previously processed
video in the first export and then in the second export we will resume
processing remaining video. For the second export we will have to
set initial timestamp offset so that video samples from the second
export are in continuation to the previously muxed samples.
PiperOrigin-RevId: 561689651
For pause and resume we need to perform multiple intermediate exports.
So moved start() logic into a separate so that it can be reused from
different places with different Compositions and MuxerWrapper.
PiperOrigin-RevId: 561675472
Switch to SparseArray<List> rather than a Map<Integer, List>.
Track indices now determine sample list dumping, reducing flakiness.
PiperOrigin-RevId: 561658586
For pause/resume feature, same MuxerWrapper needs to be
used across intermediate exports. So pass the MuxerWrapper
from Transformer.java
More specifically, when resume() is called
1. Create a MuxerWrapper and remux the previous
video samples (Export 1).
2. User the same MuxerWrapper and start processing
remaining video samples (Export 2).
PiperOrigin-RevId: 561325867
More specifically, this CL
- Defines a VideoGraph interface
- Adds a factory method, to hide the constructors
- Separate out an initialize method that does the real work (following that constructor should not do real work)
- The VideoGraph takes in a list of composition effects. For now, we only use the `Presentation`, if there is one in the list. This means we can take any number of Presentations now, but only the first one will be used.
- Moves the VideoFrameProcessingWrapper to its own file
PiperOrigin-RevId: 561059653
For pause and resume feature we need to find the timestamp of
the last sync sample in an MP4 file. The Mp4ExtractorWrapper
provides an easy to use API on top of Mp4Extractor.
PiperOrigin-RevId: 560113112
The first call to method `registerInputStream` doesn't block.
Later successive calls to the method blocks until the previous register call
finishes.
PiperOrigin-RevId: 559694490
Covers all permutations&combinations of:
* AV asset.
* AV asset with effects.
* AV asset with silence.
* AV asset with silence and effects.
Note that the video is not relevant (therefore transmuxed), but is
needed for silence generation.
PiperOrigin-RevId: 558734593
We will need to build upon original composition
for pause and resume feature and will have to set modified sequences.
In the next CL, will add method to set sequences on builder and will
probably remove it from the constructor parameter (to avoid providing
two APIs for same thing).
PiperOrigin-RevId: 558127334
I wasn't able to reproduce the error in the bug by running videoTranscoding_withTextureInput_completesWithCorrectFrameCountAndDuration
100 times, but I figured this added debug info may help the next time the
issue is seen.
PiperOrigin-RevId: 558118443
In addition to this being how Alpha should be handled, tests do fail under
API 29 without this CL. Unfortunately while these tests do fail under API 29
without this CL, we currently only run this test on API 33 emulators, so we
didn't catch this failure earlier (until compositor tests on mh failed on all
devices under API 29 and succeeded on all at or over 29).
PiperOrigin-RevId: 557781757
Rename AudioMixerImpl to DefaultAudioMixer.
Removes the AudioMixerImpl specific getOutputAudioFormat, as the caller
defines and sets this.
PiperOrigin-RevId: 555887722
To ensure that for each output bitmap from the compostor, the right input
timestamps were used.
Only applied on a subset of tests to avoid needing to upload+maintain
too many files/size in the test binary, especially when it would test
duplicate behavior
PiperOrigin-RevId: 555222530
This means we now require 2+ input frames per input, and compare the primary
stream timestamp with secondary stream timestamps in order to select the
correct output timestamp. We also must release frames and back-pressure as
soon as possible to avoid blocking upstream VFPs.
Also, improve signalling of VFP onReadyToAcceptInputFrame
PiperOrigin-RevId: 553448965
ExoPlayer queues the EOS buffer to the decoder with offset/size/timestamp all
equal to zero, and a EOS flag.
69769c77b3 set TIME_END_OF_SOURCE on the EOS buffer from the extractor.
Queueing the EOS buffer to the decoder with TIME_END_OF_SOURCE causes some
decoders to output wrong timestamps in its output.
This CL replicates what ExoPlayer does in Transformer.
PiperOrigin-RevId: 553104213
This allows for custom implementations of this interface, like
a TestVideoCompositor or partner-implemented implementation
PiperOrigin-RevId: 552541631
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).
This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)
- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
- We bypass the input manager
- The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
- The generated frame is sent to texture output and in turn saved to bitmaps
- We then compare the generated bitmap with the goldens
PiperOrigin-RevId: 551795770
Move shared test logic to the test runner.
This does increase indirection, which isn't usually preferable in tests.
However, we will have many different tests that would use logic
like this, so this allows us to reduce repetition.
PiperOrigin-RevId: 551536438
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.
PiperOrigin-RevId: 550894935
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.
Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.
Regression tests reproduce failure without fix, but pass with it.
PiperOrigin-RevId: 550853714
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.
PiperOrigin-RevId: 550839156
signalEndOfInputStream is needed for when streams have different amounts of
frames, so that if the primary stream finishes after a secondary stream, it
can end without waiting indefinitely for the secondary stream's matching
timestamps.
onEnded mirrors this API on the output side, which will be necessary to
know when to call signalEndOfInput on downstream components (ex. on downstream)
VideoFrameProcessors
PiperOrigin-RevId: 549969933
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.
Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.
PiperOrigin-RevId: 548994584
This serves as a sort of test with more frames for the compositor for now
(before more varied video system tests come later when integrating with
Transformer), showing it doesn't error out and outputs the right
amount of frames.
Due to the VFPTestRunner having a 5s timeout, and mostly due to presubmit
emulators being very slow with OpenGL, there is a sort of limitation
to how many frames this type of test can have, depending on the test target
PiperOrigin-RevId: 548159762
SingleInputVideoGraph implements GraphInput now, so the asset loaders would
interface directly with SIVG, rather than VideoSampleExporter. This is to pave
way for multi-asset video processing.
PiperOrigin-RevId: 547561042
* Allow more than one input bitmap at a time.
* Allow Compositor to take in and set up an Executor. Otherwise,
Compositor resources may be created on one thread and accessed on another.
* Add a Compositor TestRunner to reuse test code more.
* Update VideoFrameProcessingTaskExecutor to use a new onError listener, so
that it's more reusable in non-DVFP contexts, like for Compositor.
PiperOrigin-RevId: 547206053
durationUs is almost always going to be a larger number than the sample
rate, so pass as the main value, rather than the multiplier.
PiperOrigin-RevId: 547193927
TransformerAndroidTestRunner#run will throw any exception that occurs,
so there is no need to assert the result exception is null.
PiperOrigin-RevId: 546923823
Frame buffers are focused, not current. EGL Contexts may be current but this
is a prerequisite for focusing a framebuffer, and this alone isn't
sufficient to make a bitmap.
Clear this up as it can make some complex tests harder to understand, if the
methods needed don't match up in naming.
PiperOrigin-RevId: 546305118
Add documentation for threading requirements at the class level (in
addition to existing documentation on the methods) to improve
discoverablility. Also fix a couple of nits in the javadoc (US English
spelling, avoid passive voice) and in `OnInputFrameProcessedListener`.
PiperOrigin-RevId: 546303732
This allows us to simplify and reduce thread initialization, hops and the
synchronization cost associated with them, especially when more and more VFPs
are used.
PiperOrigin-RevId: 545622169
On a MediaItem change, the input Format (and Effects to apply) may be
different. Therefore the AudioProcessingPipeline must be reconfigured
to determine what processing is active, and what the AudioFormat of the
data output is. In the event that it is different, additional
AudioProcessor instances must be used to ensure the encoder will still
be able to accept the audio buffers.
PiperOrigin-RevId: 544338451
Providing the sync token in the api allows the client to decide which waiting method they would like to use depending on the use case, allowing them to optimise if possible.
PiperOrigin-RevId: 543997311
This fixes a broken javadoc link and makes it possible to customize media
source creation via the asset loader factory without also having to
implementing a decoder factory from scratch.
PiperOrigin-RevId: 543995538
Goal of tests (SequenceExportTest) that use this media is for the
silence and the media to match exactly with audio format, however
`sample_with_increasing_timestamps.mp4` had a different sample rate.
testvid_1022ms.mp4: channel count = 2, sample rate = 44100.
PiperOrigin-RevId: 543458948
MP4 edit lists sometimes ask to start playback between two samples.
If this happens, we currently change the timestamp of the first
sample to zero to trim it (e.g. to display the first frame for a
slightly shorter period of time). However, we can't do this to audio
samples are they have an inherent duration and trimming them this
way is not possible.
#minor-release
PiperOrigin-RevId: 543420218
FallbackListener.onTransformationRequestFinalized() is called from the
AssetLoader thread for audio, and from the GL thread for video.
PiperOrigin-RevId: 542851284
With the upcoming "handle format changes" CL, stereo -> mono audio
would add an AudioProcessor. Robolectric decodes output encoded data,
which crashes some AudioProcessors because the number of frames may not
be an integer.
PiperOrigin-RevId: 542568875
The effects pipeline must receive images in the sRGB colorspace due to the color transfers applied in the shaders. Currently the burden to making sure images are in the right colorspaces falls onto apps. This CL ensures that this is not the case anymore.
PiperOrigin-RevId: 542323613
Audio only tests are now using RAW audio where possible, which is
passed through the Robolectric decoders/encoders, and can be handled by
the AudioProcessor instances accurately.
PiperOrigin-RevId: 541648853
To more accurately describe what they do, especially as Compositor will
starts to use more contexts or threads, and it's important to know what
needs to be reset/recreated/focused before what methods.
PiperOrigin-RevId: 541010135
Issue: When running the Transformer related test cases, the tests are flaky
because the order in which audio and video samples are interleaved seems to
differ in few instances.
Root cause: When running a transformation the sample producer (Asset loader)
and sample consumer (Sample pipeline) both runs on different thread and
theoretically there is no reason for behaviour to be deterministic because
the number of samples produced/written depends on how fast individual thread
works. So it is indeed surprising that test somehow worked deterministically in
majority of instances (may be something to do with Robolectric environment).
Solution: Since we don't expect the order of sample interleaving to be deterministic, make the dumping logic deterministic where all the video
samples will be collected and then dumped together (similarly for audio). This would mean we won't be able to see the interleaving so for that we need to
add separate test case verifying the interleaving logic only.
Pending: Test case for interleaving.
PiperOrigin-RevId: 540930871
The existing NullableType has been deprecated 5 years ago and causes
crashes in Kotlin apps because Kotlin doesn't recognize this annotation
as a nullable type annotation.
While we can't align on a single @Nullable annotation yet, we can at
least replace this one by JSR305's @Nonnull(MAYBE) as it fulfils all
requirements, including full Kotlin compatiblity. To avoid the
cumbersome name, we can redefine it as our own @NullableType
annotation. (We can't use @Nullable to avoid name clashes with the main
@Nullable annotation from AndroidX)
Issue: google/ExoPlayer#6792
PiperOrigin-RevId: 540497469
The test transcodes four media items with distinct effects, keeping one frame
each; extracts the four frames in the produced video, and compares them against
the expected results.
PiperOrigin-RevId: 539697344
This methods currently registers and queues the bitmaps Will need to have a method that just queues bitmaps in future CLs.
#minor-release
PiperOrigin-RevId: 539605628
To ensure no regressions for the potentially confusing pipeline of:
* HDR electrical -> SDR linear EOTF+OOTF, and
* SDR linear -> SDR electrical OETF
PiperOrigin-RevId: 538741079
208eefc0fd introduced using `DefaultDecoderFactory.getDecoderInfo(format) != null` caused certain tests not to be skipped when they were expected to be, creating more mh failures.
PiperOrigin-RevId: 537820370
ExoPlayer extractors (backing `MetadataRetriever`) now parse the color format
from the bitstream so using `MetadataRetriever` should be an equivalent but
more lightweight way to verify the color info.
Also remove try/catch blocks in test code calling into these methods, and add
skipping based on decoder capabilities in the cases where it was missing.
PiperOrigin-RevId: 537789483
In some cases the codec selected for decoding has a different MIME type than
the media. In thoses cases Transformer continued to use the media's MIME type
and that caused codec configuration failures.
Removed `EncoderUtil.findCodecForFormat()` as we stopped using the method it
uses for finding a codec. Plus, the method is only used in the test.
See also `MediaCodecUtil.getALternativeCodecMimeType()`.
PiperOrigin-RevId: 536683663
