Add format codec info, which can make test skipping checks more similar to the
actual Transformer decoder checks.
Also for the test file, the actual format was 720p, but somehow the file name and
media metadata indicated 1080p. This format mismatch led to some decoding errors,
so fix the format (and associated errors). This also allows us to remove the
exception catch in ForceInterpretHdrVideoAsSdrTest, which was included due to
errors from the incorrect format.
PiperOrigin-RevId: 511809507
- Split the transmux setting into transmuxAudio and transmuxVideo. This
is more flexible for apps and will also be useful for unit testing
(particularly as we can't test video transcoding on Robolectric at the
moment).
- Move these settings to Composition. It makes sense for these settings
to be next to forceAudioTrack. Apps may also want to set these
settings based on the current Composition's MediaItems.
- Add a Composition.Builder because Composition now contains a few
optional fields.
PiperOrigin-RevId: 511708618
Logcat had the following lines, with no other information.
```
DefaultEncoderFactory: Encoders removed for resolution:
DefaultEncoderFactory: Encoders removed for bitrate:
DefaultEncoderFactory: Encoders removed for bitrate mode:
```
PiperOrigin-RevId: 511470231
Implement getMediaFormatInteger, a helper method simulating mediaformat.getInteger(name, defaultValue).
This reduces the API 29 restriction from MediaFormatUtil.getColorInfo to API 24, in
particular removing the method-based restriction to a constant-based restriction,
so that we can reduce usage of the API 29 class.
This also allows us to slightly simplify prior use-cases where we'd check
containsKey and getInteger to have a default value.
PiperOrigin-RevId: 511184301
When clipping a MediaItem with start time > 0, the audio was ending
before the video. This is because:
- Audio timestamps are computed based on the sample sizes, with a start
time set to streamOffsetUs (i.e. the streamStartPositionUs is not
taken into account).
- The SamplePipeline was subtracting streamStartPositionUs from the
timestamps before sending the samples to the muxer.
- As a result, the audio timestamps were shifted by
streamStartPositionUs, while they should be shifter by streamOffsetUs.
PiperOrigin-RevId: 511175923
* Account for `AssetLoader` output types.
* Consider cases that are not audio/video specific.
* Use `Format#sampleMimeType` for track specific conditions to check.
* Untangle `SamplePipeline` initilization from `AssetLoader` state.
PiperOrigin-RevId: 511020865
- Add silent audio when the output contains an audio track but the
current MediaItem doesn't have any audio.
- Add an audio track when generateSilentAudio is set to true.
PiperOrigin-RevId: 511005887
Previously, this was limited to API 29. Expand this to all API versions.
Also, update the test to:
(1) skip based on SDR format input instead of HDR format input
(2) Check the exception message in order to disambiguate between the decoder tone
mapping error, and general video format support error.
PiperOrigin-RevId: 511002218
Unstuck the muxer if the next timestamp in the track with the minimum
timestamp is larger than this minimum timestamp plus
MAX_TRACK_WRITE_AHEAD_US.
PiperOrigin-RevId: 510977088
Before, if the upstream AssetLoader provides HDR to the VideoSamplePipeline when
HDR_MODE_EXPERIMENTAL_FORCE_INTERPRET_HDR_AS_SDR is requested, the
VideoSamplePipeline would attempt to tell the AssetLoader to output SDR, which
could be accomplished via MediaCodec tone-mapping in the AssetLoader.
However, this makes an assumption of the AssetLoader implementation, and
AssetLoaders may not all implement support for decoder tone-mapping. Remove javadoc
attempting to explain how AssetLoaders (ex. custom ones) could behave.
PiperOrigin-RevId: 510956820
This method uses sampleRate, channelCount and pcmEncoding, so passing
AudioFormat is easier.
This will lead into a future change that builds the
encoderInputAudioFormat from encoder.getConfigurationFormat()
PiperOrigin-RevId: 510956177
Rename ScaleToFitTransformation to ScaleAndRotateTransformation.
This better represents the operations that can be accomplished using this
effect. The name was originally named ScaleToFit* because it's not obvious how
to scale to fit using OpenGL, and this effect handled the scaling to fit in a way that no other MatrixTransformations did.
However, it's hard to discover how to rotate when skimming names of effects, so
it's probably more useful to convey that this effect rotates, than that it
scales to fit.
PiperOrigin-RevId: 510480078
Format.toString unfortunately doesn't log colorInfo, and as Format holds a very
large set of values, it's unclear that it should. ColorInfo is useful for codec
exceptions though, so log this in ExportException.createForCodec.
PiperOrigin-RevId: 510475520
Otherwise, a lack of HDR decoding support will result in the tests checking output files for HDR output, like HdrEditingTest.transform_noRequestedTranscode_hdr10File_transformsOrThrows, failing.
PiperOrigin-RevId: 510213020
This allows us to fix usage of HDR_MODE_EXPERIMENTAL_FORCE_INTERPRET_HDR_AS_SDR.
Before, this was checked in the VideoSamplePipeline, which no longer decides on the decoder configuration input format.
PiperOrigin-RevId: 510142097
Some of the test methods in `TransformationTest` don't check
capabilities. Add a check just scoped to decoding (not checking
encoding, but the default encoder factory may fall back).
PiperOrigin-RevId: 510088562
* Moved the logic to SamplePipeline.
* Pass the requested values via Format.
* Moved exception throwing inside the methods.
* Build up the mimeTypesToCheck as a set - removing possible duplicate
checks.
* Simplified logic that calls the findSupportedMimeType method.
* Improved javadoc.
PiperOrigin-RevId: 509594062
It may be confusing for 3P apps, for us to have separate error codes for (1) if
encoding HDR is not supported at all by the device, and (2) if encoding the
format, which happens to be HDR, is not supported by the device. Instead, we can
communicate this in the error message.
PiperOrigin-RevId: 509188666
- For single-asset, the behavior stays the same. Transcode if and only
if it's necessary,
- For constrained multi-asset, always transcode, except if the setter to
transmux is set. This is to avoid failing if a MediaItem that doesn't
require transcoding is followed by a MediaItem that does require
transcoding.
PiperOrigin-RevId: 508097798
It doesn't actually make sense for them to be placed in the Transformer, because the error's root causes are actually only in codecs. Also, a few codec errors were
repeated, so deduplicate these instances
PiperOrigin-RevId: 506937695
Based on [this conversation thread](https://chat.google.com/room/AAAA--f88ao/76Rem_cRCK8), I've opted to update the existing FrameProcessor.create() rather than deprecate it, as it is unlikely to be in use by apps outside google3.
PiperOrigin-RevId: 506920930
* Overload added `(cause, errorCode, isVideo, isDecoder, details)`,
where `details` is a string of values to be added to the error message
of the `TransformationException`.
* Overload with `MediaFormat` and `mediaCodecName` moved to
`DefaultCodec`, because all usages of that overload were from
`DefaultCodec`, and this allows a simplified API because of internally
stored values.
* `mediaCodecName` removed from overload that takes a `Format`.
* Reordered `createForCodec` parameters.
PiperOrigin-RevId: 506895268
In encoding small and odd-numbered resolutions, like `316x61` ([this image](https://upload.wikimedia.org/wikipedia/commons/6/65/100winners.png)), the current fallback logic prefers a software encoder to hardware ones. The assumption was, the encoder factory applies the encoder size alignment and changes the resolution to `316x60` for SW encoders and `316x64` for HW ones. SW encoders is selected because the supported resolution 60 is closer to requested 61, than the hardware supported 64.
This change changes the default encoder selection process to only expose hardware encoders if there is any.
PiperOrigin-RevId: 506879983
With multi-asset, the sample pipelines can process more than one
MediaItem. The renaming makes it clear that the format passed to the
SamplePipeline constructors is the one corresponding to the first
MediaItem. Indeed, the first format is the one used to configure the
SamplePipelines.
PiperOrigin-RevId: 506879260
The prior version (with the call to createEncodingException) could
never occur as select...SupportedMimeType already checks for HDR
editing support. This change ensures we throw before creating an
encoder, gives a better error code and allows future simplifications
around createForCodec (see child CL).
PiperOrigin-RevId: 506308290
Transformer callbacks will take a Composition instead of a MediaItem.
Apps should be able to see what this Composition contains.
PiperOrigin-RevId: 505976561
Add checks to GL tone-mapping pixel tests, to ensure the device's decoder, API
version, and OpenGL implementation support GL tone-mapping before attempting it.
These tests should be run on mobile harness, to detect per-device failures, and
so are moved to transforemr/mh. Per b/263395272, these tests should ultimately
be in an effect/mh directory.
PiperOrigin-RevId: 505749974
ParcelFileDescriptor is quite specific and removing it allows to have
less overloads of startTransformation.
A follow-up CL will undeprecate the overload that takes a MediaItem.
PiperOrigin-RevId: 505670321
Also add a Builder to EditedMediaItem to avoid having a constructor with
many optional parameters, or a chain of constructors.
PiperOrigin-RevId: 504588544
Transformer convention has been to avoid splitting into packages unless
necessary.
See https://jlbp.dev/JLBP-2: "Prefer fewer packages over more packages
to avoid unnecessarily publicizing internal details, since any
dependency across package boundaries needs to be public.
PiperOrigin-RevId: 503463294
This is an AssetLoader that wraps a sequence of AssetLoaders. It will
be used for constrained multi-asset.
This class can currently only concatenate media items with the exact
same format.
PiperOrigin-RevId: 502525796
Also remove usages of TransformationRequest convenience methods
(setScale, setRotationDegrees and setResolution).
Some usages of setResolution can't be removed yet because they are used
for fallback.
PiperOrigin-RevId: 502415748
This is needed for constrained multi-asset to shift the timestamps of
the media items that are not the first in the sequence.
PiperOrigin-RevId: 502409923
This class may be removed soon, but in the meantime clarify this method's
javadoc summary fragment to make it marginally clearer that this only affects
the "displayed" height instead of the "encoded" height.
PiperOrigin-RevId: 501582219
Small refactoring change to simplify the logic a tiny bit by removing one
unnecessary variable. No functional change intended.
PiperOrigin-RevId: 501566533
It makes more sense to have it in the AssetLoader now that decoding is
done there, particularly as the decoder factory should rarely be
customized.
Also remove the setter from the AssetLoader.Factory because it's
uncommon to have setters in a factory.
PiperOrigin-RevId: 501244974
This brings together the multiple details about a muxer track, and
reduces the need for additional variables and more complicated track
tracking.
PiperOrigin-RevId: 499872145
The old names are not really correct anymore because:
- The Audio/VideoTranscodingSamplePipelines do not decode anymore.
- The pipelines now mux the encoded data.
PiperOrigin-RevId: 499498446
Whilst testing fallback functionality, I found that we were
aggressively reducing the resolution if it was not supported. A quick
test found that we could reduce by a much smaller increments.
Performance wise it appears these checks are incredibly quick.
The code for checking supported sizes was duplicated, with one case
having a bug because of this duplication (2/3 case). This CL abstracts
this into a loop.
PiperOrigin-RevId: 499497646
When a listener method is deprecated, the new method should (by
default) called through to the deprecated one. This is because any
class that implements the method that is now deprecated needs to still
receive that callback.
It appears when onTransformationError(MediaItem, Exception) was
deprecated in favour of onTransformationError(MediaItem,
TransformationException), this deprecation was the wrong way round, and
the newer callback - onTransformationError(MediaItem,
TransformationResult, TransformationException) continued this mistake.
This CL now corrects this.
PiperOrigin-RevId: 498221504
Events on the wrapper should be propagated to TransformerInternal as
soon as they occur, switching round the process so TransformerInternal
does not have to query MuxerWrapper.
This CL is a prerequisite for the child CL, where MuxerWrapper can
simplify the internal state and logic.
PiperOrigin-RevId: 497267202
Before, it was possible for onDurationUs() and onAllTracksRegistered()
to be called before onTrackAdded() because they are called from
different threads. onDurationUs() and onAllTracksRegistered() are called
from the Transformer internal thread, and onTrackAdded() is called from
the playback thread.
PiperOrigin-RevId: 497102556
AssetLoader declares the tracks with a setTrackCount() method. Setting
the track count on the MuxerWrapper is easier than calling
registerTrack() as many times as the number of tracks.
PiperOrigin-RevId: 496933501
AssetLoader declares the tracks with a setTrackCount() method. Setting
the track count on the FallbackListener is easier than calling
registerTrack() as many times as the number of tracks.
PiperOrigin-RevId: 496919969
The TransformationRequest passed to FallbackListener (and
createSupportedTransformationRequest) can have null for values that
are inferred from source. Within fallback, this can be height, width,
video mime type and audio mime type (HDR mode is not linked to source).
requestedFormat has these values populated from source, and
supportedFormat then finds the closest supported values to the
requested.
If any of the values in supportedFormat do not match the
requestedFormat, then this method would build upon the
TransformationRequest and update ALL possible fallback fields. This is
a problem because the fallback listener compares the original request
to the fallback one and notifies about all the fields that have
changed.
This CL changes this so that only the values that are not the same as
requested are changed in the supported request that is given to the
fallback listener.
PiperOrigin-RevId: 496908492
When trying to run the test on Android Studio, error "incompatible
types: Buffer cannot be converted to ByteBuffer" is logged. This is
because ByteBuffer.flip() returns a Buffer (and not a ByteBuffer).
Annotation @CovariantReturnType on ByteBuffer.flip() should resolve this
automatically but it doesn't seem supported at the moment.
PiperOrigin-RevId: 496894723
Note that we simply use GlEffectsFrameProcessor in-app / GL tone-mapping, so PQ->SDR tone-mapping isn't yet implemented.
Tested manually using the demo on Pixel 7, to confirm that device and in-app tone
mapping behave similarly.
PiperOrigin-RevId: 496700231
Adds the AudioMixerAlgorithm interface which allows for specialized
implementations of audio mixing that also efficiently convert between
source and mixing formats.
Initial implementation has two algorithms:
1. Float -> float (with channel mixing)
2. S16 -> float (with channel mixing)
PiperOrigin-RevId: 496686805
To do that, rename PROGRESS_STATE_NO_TRANSFORMATION to
PROGRESS_STATE_NOT_STARTED and update Javadoc of ProgressState to not be
Transformer specific.
PiperOrigin-RevId: 496653460
This should allow us to focus on HDR failures instead of network buffering
failures when debugging HDR issues.
These files are each used on several files, so it should be more worth the
test binary impact to move these files to local first.
Locally, tests did take less time after this diff
PiperOrigin-RevId: 496398130
The MediaSourceFactory won't be used by the other AssetLoaders
In order to do that, ExoPlayerAssetLoader has been made public, and the
DefaultAssetLoaderFactory has become a wrapper around
ExoPlayerAssetLoader.
PiperOrigin-RevId: 496386853
Otherwise, the decoders are not captured. It works at the moment for the
video decoder because decoding is still done on the sample pipeline but
it will moved to the AssetLoader soon.
PiperOrigin-RevId: 495275575
`AudioProcessor`s expect direct buffers. This shouldn't make any functional difference in our code, but a custom audio processor might try to access the buffer from JNI in which case a direct byte buffer is more efficient.
PiperOrigin-RevId: 495241669
`SilentAudioGenerator` could output a fractional audio frame, and this could cause downstream components to throw because of trying to read a complete audio frame but only seeing a partial one.
Calculate the output buffer size based on the frame size (which is a no-op for stereo 16-bit audio) and calculate a total number of frames to output then multiple by the frame size.
PiperOrigin-RevId: 494992941
The duration in TransformerInternal.ComponentListener is set on the
Transformer internal thread, and is read on the playback thread. Making
this field volatile ensures that the playback thread reads the updated
value.
PiperOrigin-RevId: 493908385
Rename ERROR_CODE_OUTPUT_FORMAT_UNSUPPORTED to
ERROR_CODE_ENCODING_FORMAT_UNSUPPORTED.
This makes the error code more consistent with ERROR_CODE_DECODING_FAILED on the
decoding side. Also, the error code is in the "Encoding errors (4xxx)" section,
so muxer errors probably should be in the "Muxer errors (7xxx)" section instead.
Additionally, no muxer errors currently seem to use
ERROR_CODE_OUTPUT_FORMAT_UNSUPPORTED, so this should be a safe change.
PiperOrigin-RevId: 493322880
This is necessary to move video slow motion flattening to the
AssetLoader because this step can change the duration. As we use the
duration before flattening to calculate the progress, we must also use
the position before flattening.
PiperOrigin-RevId: 493291990
Modify FrameProcessor and MatrixTextureProcessor interfaces to support
different input and output color transfers. Does not implement conversion between
color ranges (ex. HDR and SDR), but should allow for conversion between color
transfers of the same color range (ex. HLG and PQ).
This supports in-app tone mapping, where we need a single FrameProcessor to
input HDR color transfers (ex. HLG/PQ) and output SDR (ex. gamma2.2). This also
supports previewing, where we need a single FrameProcessor to be able to input HLG
and output PQ.
Manually tested by confirming colors still look right on SDR and HDR videos
with a rotation and color affect applied.
PiperOrigin-RevId: 493108678
From this CL on, FrameProcessor listeners will be invoked from an Executor that
is passed in when creating the FrameProcessor.
GlTextureProcessor needs to invoke the ErrorListener on the said Executor too.
PiperOrigin-RevId: 493018583
This is necessary to move video decoding to the AssetLoader. Otherwise,
if the decoder max pending frame count is reached, the AssetLoader will
stop queuing frames to the pipeline, and process data will not be called
anymore.
PiperOrigin-RevId: 492392621
EncoderUtil using a static list to cache encoders means it's not possible to
use a different set of encoders for different tests when running all
robolectric tests together.
PiperOrigin-RevId: 491959350
Bypass Z Fold 4 HDR10 tone-mapping bug by limiting the max frame count to 12.
This passed with a value of 14, and failed with a value of 15, but I figured I'd use 12 just to be safe.
PiperOrigin-RevId: 491684058
Main change:
- Removed `Codec.EncoderFactory.createForVideoEncoding`'s argument of a list
of allowed MIME types
- Moved the check for whether a video MIME type is supported to VTSP
PiperOrigin-RevId: 491611799
This thread just starts the player and handles the player callbacks for
now. Sample pipelines are still run on the playback thread.
PiperOrigin-RevId: 491299671
This reduces the priority to best effort (from the default that seems to be best effort), and allows us to run SSIM even on 8k24fps video. Without this CL,
start()'ing a second codec may result in a MediaCodec.CodecException.
Tested to confirm that transformation8k24():
* fails deterministically without this CL, or with KEY_PRIORITY set to 0.
* succeeds deterministically after this CL (~18s on Samsung Z Fold 4)
PiperOrigin-RevId: 490570416
We cannot check this in code, due to DEVICE_INITIAL_SDK_INT being a @SystemApi, and
reflection being a bit risky/unstable.
PiperOrigin-RevId: 490537916
The bug has since been fixed.
The values still could change, as the API is labelled as @UnstableApi, so it's
probably fine to leave the <p> tag mostly as is.
PiperOrigin-RevId: 490509205
This allows us to release both codecs used in SSIM when one fails to
configure() or start().
Tested and confirmed that on Samsung Galaxy Z Flip 4, running all
TransformationTest.java tests, tests after transform8k24() fails to start the
2nd codec:
* Before this CL, all fail.
* After this CL, all pass.
PiperOrigin-RevId: 490461560
