* Non-standard parameter comment; prefer `/* paramName= */ arg`
(see http://go/bugpattern/ParameterComment) (3 times)
This CL looks good? Just LGTM and Approve it!
This CL doesn’t look good? This is what you can do:
* Revert this CL, by replying "REVERT: <provide reason>"
* File a bug under go/error-prone-bug for category ErrorProneStyle if there's an issue with the CL content.
* File a bug under go/rosie-bug if there's an issue with how the CL was managed.
* Revert this CL and not get a CL that cleans up these paths in the future by
replying "BLOCKLIST: <provide reason>". This is not reversible! We recommend to
opt out the respective paths in your CL Robot configuration instead:
go/clrobot-opt-out.
This CL was generated by CL Robot - a tool that cleans up code findings
(go/clrobot). The affected code paths have been enabled for CL Robot in //depot/google3/java/com/google/android/libraries/media/METADATA which is reachable following include_presubmits from //depot/google3/third_party/java_src/android_libs/media/METADATA.
Anything wrong with the signup? File a bug at go/clrobot-bug.
#codehealth
Tested:
Local presubmit tests passed.
PiperOrigin-RevId: 472254253
shouldPassthrough's internal checks seem to be check whether we should *not*
pass through, which seemed a bit like a confusing double-negative to me.
shouldTranscode is slightly more clear, by instead returning true when we do
want to transcode.
No functional changes intended.
PiperOrigin-RevId: 471753771
3b0d2c1586 made `shouldPassthrough` always return false for `enableHdrVideoEditing`:
>We force using `FrameEditor` (no passthrough) to avoid the need to select another edit operation, and use the new shaders. The `EGLContext` and `EGLSurface` also need to be set up differently for this path.
However, this was introduced before the `videoNeedsEncoding` setting was introduced in 3f615040c0. That setting should apply to HDR videos as much as SDR videos.
PiperOrigin-RevId: 471569853
* Non-standard parameter comment; prefer `/* paramName= */ arg`
(see http://go/bugpattern/ParameterComment) (3 times)
This CL looks good? Just LGTM and Approve it!
This CL doesn’t look good? This is what you can do:
* Revert this CL, by replying "REVERT: <provide reason>"
* File a bug under go/error-prone-bug for category ErrorProneStyle if there's an issue with the CL content.
* File a bug under go/rosie-bug if there's an issue with how the CL was managed.
* Revert this CL and not get a CL that cleans up these paths in the future by
replying "BLOCKLIST: <provide reason>". This is not reversible! We recommend to
opt out the respective paths in your CL Robot configuration instead:
go/clrobot-opt-out.
This CL was generated by CL Robot - a tool that cleans up code findings
(go/clrobot). The affected code paths have been enabled for CL Robot in //depot/google3/java/com/google/android/libraries/media/METADATA which is reachable following include_presubmits from //depot/google3/third_party/java_src/android_libs/media/METADATA.
Anything wrong with the signup? File a bug at go/clrobot-bug.
#codehealth
Tested:
Local presubmit tests passed.
PiperOrigin-RevId: 471198016
In particular, make it a bit more clear that "rendering" and "releasing" frames are
related concepts, and how they differ from one another in conjunction with frame
dropping.
PiperOrigin-RevId: 471037733
* Non-standard parameter comment; prefer `/* paramName= */ arg`
(see http://go/bugpattern/ParameterComment) (19 times)
This CL looks good? Just LGTM and Approve it!
This CL doesn’t look good? This is what you can do:
* Revert this CL, by replying "REVERT: <provide reason>"
* File a bug under go/error-prone-bug for category ErrorProneStyle if there's an issue with the CL content.
* File a bug under go/rosie-bug if there's an issue with how the CL was managed.
* Revert this CL and not get a CL that cleans up these paths in the future by
replying "BLOCKLIST: <provide reason>". This is not reversible! We recommend to
opt out the respective paths in your CL Robot configuration instead:
go/clrobot-opt-out.
This CL was generated by CL Robot - a tool that cleans up code findings
(go/clrobot). The affected code paths have been enabled for CL Robot in //depot/google3/java/com/google/android/libraries/media/METADATA which is reachable following include_presubmits from //depot/google3/third_party/java_src/android_libs/media/METADATA.
Anything wrong with the signup? File a bug at go/clrobot-bug.
#codehealth
Tested:
Local presubmit tests passed.
PiperOrigin-RevId: 471022923
Use the PQ OETF and EOTF to ensure that intermediate fragment shader operations
using PQ are in linear BT.2020 rather than PQ and HLG-1 BT.2020.
Also, swap the OETF and EOTF in shaders, as they were used incorrectly before
Manually tested by verifying transformer demo HLG and PQ videos look the same with and without this CL, including with a BitmapOverlayProcessor enabled to test flows both with one MatrixTransformationProcessor that skips HDR TFs, and with one that doesn't.
PiperOrigin-RevId: 469736067
Adds a method to FrameProcessor.Listener to be called when an
output frame is available and a method releaseOutputFrame in
FrameProcessor allowing the caller to trigger release of the
oldest available output frame at a given timestamp. Late frames
or frames with unset release times are dropped in the
FinalMatrixTransformationProcessorWrapper.
More than one output frame can become available before they are
released if the penultimate GlTextureProcessor is capable of producing
multiple output frames. Processing continues while waiting for
releaseOutputFrame to be called. Frame release tasks are prioritized
over other tasks.
PiperOrigin-RevId: 468473072
We create an empty CueGroup in many places as default or
where none is needed. Instead, we can define a constant
for this purpose and reuse it.
PiperOrigin-RevId: 467944841
Increase the estimated max sample size for HEVC by 2x, and set a minimum
size of 2MB. The 2MB will be applied for resolutions up to 1080p, after
which the new calculation takes effect. This is in par with the
platform's HEVC software decoder.
PiperOrigin-RevId: 467641494
This base class will simplify the implementation of custom
Player classes. The current version only supports
available commands and playWhenReady handling.
PiperOrigin-RevId: 467618021
createSupportedTransformationRequest is more accurate than
createFallbackTransformationRequest, as a TransformationRequest will be returned
regardless of whether any fallback is applied.
PiperOrigin-RevId: 466641277
Previously, this feature interpreted SDR signals as HDR when called. Now, only HDR
streams are interpreted as HDR, so the javadoc should be updated. Not yet removing
this method, as there are still some loose ends to finish up (ex. PQ support, e2e
tests).
PiperOrigin-RevId: 466425738
While HDR is most closely tied to the color transfer (ex.
COLOR_TRANSFER_SDR is the only one explicitly mentioning dynamic
range), technically color spaces may be associated with HDR as well,
like BT.2020 commonly being used for HDR rather than BT.709 for SDR.
Therefore, it's more specific to mention just that the transfer is HDR.
PiperOrigin-RevId: 466316960
This allows the GlEffectsFrameProcessor to later handle HLG and PQ
differently, or limited and full color range differently.
No functional change intended in this CL.
PiperOrigin-RevId: 466070764
In the case where this check fails, the downstream frame processor chain won't be able to handle the incoming (SDR) data anyway as we've already set it up for HDR.
PiperOrigin-RevId: 465584814
`requestCalculateSsim` more clearly represents the intention of the caller.
Also rephrase the javadoc to simplify it and make it more precise.
PiperOrigin-RevId: 465575578
Adds a new event to AudioOffloadListener to get the offload state of the track, which indicates when software decoding is taking place.
PiperOrigin-RevId: 465264362
Avoids disabling Offload on a write error, and instead relies on this being disabled on the AudioTrack init. It will no longer recover by disabling offload.
PiperOrigin-RevId: 465248917
Format.NO_VALUE is a placeholder value for an invalid @C.ColorTransfer, used
for example when the decoder doesn't support this transfer function.
When encountering this invalid value, interpret this as COLOR_TRANSFER_SDR.
Confirmed locally that an exception is thrown when transcoding on p4head, and no exception is thrown when transcoding with this CL.
PiperOrigin-RevId: 464135080
This will allow effects preview in ExoPlayer to use the
Effect and FrameProcessor interface (and the interfaces
they depend on) without depending on transformer or the
future effects module.
PiperOrigin-RevId: 464060047
* Sets KEY_HDR_STATIC_INFO from MediaFormat in the DefaultCodec.
* Adds checks in mediaparser to ensure color space, range, and transfer are valid
values.
PiperOrigin-RevId: 463921325
Despite unregistering the callback and clearing pending Handler
messages, the callback may still receive pending calls if they
are already triggered by the AudioTrack. Instead of asserting
that the track is correct, we should gracefully ignore stale
events.
PiperOrigin-RevId: 463851393
TransformationException error codes were previously tied to OpenGL
but other FrameProcessor implementations are possible. So this CL
renames the error codes.
Also, remove GL_INIT_FAILED error code, as FrameProcessor
implemenations may initialize resources on a background thread
after the factory method returns, so it's not obvious how to
distinguish between initialization failures and processing failures.
PiperOrigin-RevId: 463704902
Just a misc nit, since I found the name a bit confusing, and figured
findEncoderWithClosestSupportedFormat might be more descriptive.
PiperOrigin-RevId: 463433646
Extract a FrameProcessor.Factory interface from GlEffectsFrameProcessor
and allow it to be customized using a setter on Transformer.Builder.
PiperOrigin-RevId: 463433438
Size requires API 21. Using Pair instead will allow effects to be
used from API 18 during previewing once they are moved out of
transformer.
PiperOrigin-RevId: 463206474
Add support for multi-value tags based on null terminators.
These are specific to ID3v2.4, but is backwards compatible with
ID3v2.3, so no version checks are needed.
This is needed for applying effects to a playlist.
The effects are applied based on the presentation time of the
frame in its corresponding media item and the offset is added
back before encoding.
Each time the offset changes, end of input stream is signalled
to the texture processors. This is needed because the texture
processors can expect monotonically increasing timestamp within
the same input stream but when the offset changes, the timstamps
jump back to 0.
PiperOrigin-RevId: 462714966
This allows us to use BT.2020 RGB linear for intermediate shaders, which also
allows us to re-enable PeriodicVignetteProcessor, which should work properly in
linear color-spaces.
Manually tested by adding a GlEffectsWrapper, and confirming that HLG HDR editing still looks correct.
PiperOrigin-RevId: 462265821
This adds two methods that are helpful when working with Futures.
One is a version of postOrRun that can indicate completion by a
Future and the other is a simplified version of Guava's
Futures.transformAsync (which can't be used as it's in Beta).
PiperOrigin-RevId: 461896598
Otherwise, invalid ColorInfo instances generated using faulty
MediaFormat#getInteger values could cause exceptions.
Confirmed that b/239435670 reproduces without this CL, and does not reproduce
with this CL.
PiperOrigin-RevId: 461862191
This will help developers self-diagnose issues like Issue: google/ExoPlayer#10392
where the NPE occurs far from the original null value because a field
gets assigned to null.
This change aims to ensure that every stable method on Player,
ExoPlayer and ExoPlayer.Builder that takes a non-null type will fail
with an NPE before returning.
#minor-release
PiperOrigin-RevId: 461846580
* Introduced `useHdr` for `GlEffect#toGlTextureProcessor`, so
`TextureProcessor` implementations can decide how to handle HDR.
* Creating FP16 color textures for HDR input.
Tested via manual testing, adding a no-op GlEffectWrapper to the transformation to
force use of intermediate textures, adding a linear ramp to the fragment shader,
and trying to ascertain that there's a real reduction in posterization when
switching from 4-bit to 8-bit unsigned bytes, and again from 8-bit unsigned bytes
to 16-bit floating point.
PiperOrigin-RevId: 461613117
This may happen when a containers' color transfer incorrectly does not match
the video's color transfer.
An example of a file with such a mismatch is the current Transformer demo HDR10
sample file.
Manually tested by confirming that no errors are emitted for SDR and HLG sample
files, and that errors are emitted for our incorrect HDR10 sample file.
PiperOrigin-RevId: 461583532
- Update profile selection logic to pick an HDR-compatible profile when doing HDR editing on H.264/AVC videos.
- Handle doing the capabilities check for all MIME types that support HDR (not just H.265/HEVC).
- Fix a bug where we would pass an HDR input color format to the encoder when using tone-mapping.
- Tweak how `EncoderWrapper` works so decisions at made at construction time.
Manually tested cases:
- Transformation of an SDR video.
- Transformation of an HDR video to AVC (which triggers fallback/tone-mapping on a device that doesn't support HDR editing for AVC).
- Transformation of an HDR video with HDR editing.
PiperOrigin-RevId: 461572973
Also remove VideoEncoderSettings.colorProfile as there are no
concrete use cases for customizing this and it clashes with picking
the color format automatically based on SDR vs. HDR.
PiperOrigin-RevId: 460746987
The call doesn't currently reset the already loaded suppliers and
factories. Also fix the supplier loading code to use a local copy
of the current dataSourceFactory to avoid leaking an updated
instance to a later invocation.
Issue: androidx/media#116
#minor-release
PiperOrigin-RevId: 460721541
We currently start a simple Thread to release AudioTracks
asynchronously. If many AudioTracks are released at the same
time, this may lead to OOM situations because we attempt to
create multiple new threads.
This can be improved by using a shared SingleThreadExecutor.
In the simple case of one simmultaneous release, it's exactly
the same behavior as before: create a thread and release it
as soon as it's done. For multiple simultanous releases we
get the advantage of sharing a single thread to avoid creating
more than one at the same time.
Issue: google/ExoPlayer#10057
PiperOrigin-RevId: 460698942
`HevcConfig.parse` misreads reserved bit to determine NAL unit type. This is currently meant to be always set to 0, but could be given some kind of meaning in a future revision.
Issue: google/ExoPlayer#10366
PiperOrigin-RevId: 460487613
Some calls to handleBuffer return false while a previous
flush is still handled in the background.
Fix this by either asserting the method returns true if
we don't expect any delay, or calling it repeatedly until
it returns true (within a timeout).
PiperOrigin-RevId: 460474419
The call doesn't currently reset the already loaded suppliers and
factories. Also fix the supplier loading code to use a local copy
of the current dataSourceFactory to avoid leaking an updated
instance to a later invocation.
Issue: androidx/media#116
#minor-release
PiperOrigin-RevId: 460721541
(cherry picked from commit 6be0d6ea25)
I don't think it's useful to keep these in numerical order, it makes
more sense to keep them grouped into a 'logical' ordering.
#minor-release
PiperOrigin-RevId: 460453464
This extension is needed for editing HDR input with OpenGL, as the
ExternalTextureProcessor samples raw YUV values from the
external texture for HDR and converts them to RGB itself rather than
relying on the OpenGL driver to do this automatically as for SDR.
PiperOrigin-RevId: 460424154
This saves an intermediate texture copy step for use-cases
where matrix transformations are the first or only effects
in the chain.
PiperOrigin-RevId: 460239403
- Added setter to disable this feature.
- Added accompanying tests.
- Plan to run tests on the same set of settings on H265.
PiperOrigin-RevId: 460238673
I don't think it's useful to keep these in numerical order, it makes
more sense to keep them grouped into a 'logical' ordering.
#minor-release
PiperOrigin-RevId: 460453464
(cherry picked from commit 1b1e0f890d)
We wait until a previous AudioTrack has been released before
creating a new one. This is currently done with a thread
block operation, which may cause ANRs in the extreme case
when someone attempts to release the player while this is
still blocked.
The problem can be avoided by just returning false from
DefaultAudioSink.handleBuffer to try again until the previous
AudioTrack is released.
Reproduction steps to force the issue:
1. Add Thread.sleep(10000); to the AudioTrack release thread.
2. Add this to the demo app:
private int positionMs = 0;
Handler handler = new Handler();
handler.post(new Runnable() {
@Override
public void run() {
player.seekTo(positionMs++);
if (positionMs == 10) {
player.release();
} else {
handler.postDelayed(this, 1000);
}
}
3. Observe Player release timeout exception.
These steps can't be easily captured in a unit test as we can't
artifically delay the AudioTrack release from the test.
Issue: google/ExoPlayer#10057
PiperOrigin-RevId: 459468912
The SDR constant also specified a color space and range, in addition to
C.COLOR_TRANSFER_SDR. However, it turns out that SDR videos may use different color
space and range values, so following prior ExoPlayer conventions to have `null`
mean "generic SDR" is preferable here.
PiperOrigin-RevId: 459296746
If the input is HDR (HLG), check encoder capabilities for HDR support
and request tone-mapping to SDR during decoder configuration otherwise.
Capabilities are only checked for API 31 and above, as HDR editing is
not supported before.
As the encoder capabilities check needs to happen before selecting the
encoder to use (as this may depend on the resolution output by the
effects chain), the EncoderWrapper checks all candidate encoders
for the MIME type for HDR capabilities and only requests fallback to
SDR if none of them support it.
When the actual encoder is selected, the wrapper checks that it matches
one of the encoders is checked capabilities for.
PiperOrigin-RevId: 458511599
Configure the GL shaders and encoder to take in HDR metadata.
This mostly just consists of passing the Format.colorInfo through
the VideoTranscodingSamplePipeline down to the encoder, rather than passing
the PQ-ness down to the GL step.
Due to b/237674316, this will remove HDR10+ support temporarily to introduce
support for HLG10.
Manually tested to confirm that HLG10 operations that don't affect color display
correctly after this CL with "HDR editing" in the demo checked, and continue to display incorrectly (as before this CL) without the option unchecked.
PiperOrigin-RevId: 458490810
ProgressiveMediaPeriod loads all available tracks into SampleStreams
(because it needs to read the data anyway and it allows easy activation
of tracks without reloading). However, the SampleStreams for disabled
tracks are not read and no one if waiting for them.
The buffered position is used for user-visible state (e.g. in the UI)
and to check how much data is already buffered to decide when to stop
buffering (using LoadControl). Both values benefit from only
using the actually enabled tracks to better reflect what is available
for playback at the moment.
Issue:Issue: google/ExoPlayer#10361
PiperOrigin-RevId: 458475038
We used "ALL_COOECS" previously, and it is not necessary because "ALL_CODECS"
additionally the codecs that support tunneling/secure decoding, which there
is no use case in Transformer.
PiperOrigin-RevId: 458470278
Although MediaCodec claims supporting float frame rate, encoder init failed on
API21 Nexus 5. Since it's just a performance hint to the codec, it's OK to
generalize it to other API versions.
PiperOrigin-RevId: 458434650
As per MP4 spec, bitrates in esds boxes can be a 32 bit number which doesn't fits in Java int type, so now reading it as a long value. Our class for holding media format, only allows bitrates value to be an int as we don't expect the bitrates to be greater than or equal to 2^31. So we're limiting the values for bitrates to Integer.MAX_VALUE.
#minor-release
PiperOrigin-RevId: 458423162
The GlEffectsFrameProcessor that will be part of the effects module
uses the DebugViewProvider. So it does not make sense for it
to be an inner interface of Transformer.
PiperOrigin-RevId: 458014932
The FinalMatrixTransformationProcessorWrapper ensures that the
surface is only replaced when it is not being rendered to and vice
versa.
PiperOrigin-RevId: 458007639
The outputHeight in the TransformationRequest is the height of
the frame as it would be displayed (i.e., after applying any
rotation specified in the format). So pass-through should only
be used if the requested outputHeight matches the input
format's height after applying the rotation.
PiperOrigin-RevId: 457934867
Previously two timelines that differed only in shuffle order were
considered equal, which resulted in no call to
Player.Listener.onTimelineChanged when calling
ExoPlayer.setShuffleOrder. This in turn resulted in no call to
MediaControllerCompat.Callback.onQueueChanged.
Also make a small fix inside ExoPlayerImpl.setShuffleOrder, to ensure
that the new shuffle order is used when constructing the masked
timeline.
Issue: google/ExoPlayer#9889
#minor-release
PiperOrigin-RevId: 457703727
NoUidTimeline still exists as a private detail of TestUtil, but it no
longer extends ForwardingTimeline because the interactions are quite
hard to reason about.
#minor-release
PiperOrigin-RevId: 457703593
We wait until a previous AudioTrack has been released before
creating a new one. This is currently done with a thread
block operation, which may cause ANRs in the extreme case
when someone attempts to release the player while this is
still blocked.
The problem can be avoided by just returning false from
DefaultAudioSink.handleBuffer to try again until the previous
AudioTrack is released.
Reproduction steps to force the issue:
1. Add Thread.sleep(10000); to the AudioTrack release thread.
2. Add this to the demo app:
private int positionMs = 0;
Handler handler = new Handler();
handler.post(new Runnable() {
@Override
public void run() {
player.seekTo(positionMs++);
if (positionMs == 10) {
player.release();
} else {
handler.postDelayed(this, 1000);
}
}
3. Observe Player release timeout exception.
These steps can't be easily captured in a unit test as we can't
artifically delay the AudioTrack release from the test.
Issue: google/ExoPlayer#10057
PiperOrigin-RevId: 459468912
(cherry picked from commit a80dd60449)
ProgressiveMediaPeriod loads all available tracks into SampleStreams
(because it needs to read the data anyway and it allows easy activation
of tracks without reloading). However, the SampleStreams for disabled
tracks are not read and no one if waiting for them.
The buffered position is used for user-visible state (e.g. in the UI)
and to check how much data is already buffered to decide when to stop
buffering (using LoadControl). Both values benefit from only
using the actually enabled tracks to better reflect what is available
for playback at the moment.
Issue:Issue: google/ExoPlayer#10361
PiperOrigin-RevId: 458475038
(cherry picked from commit 577e19168d)
As per MP4 spec, bitrates in esds boxes can be a 32 bit number which doesn't fits in Java int type, so now reading it as a long value. Our class for holding media format, only allows bitrates value to be an int as we don't expect the bitrates to be greater than or equal to 2^31. So we're limiting the values for bitrates to Integer.MAX_VALUE.
#minor-release
PiperOrigin-RevId: 458423162
(cherry picked from commit 9e10286b5e)
Previously two timelines that differed only in shuffle order were
considered equal, which resulted in no call to
Player.Listener.onTimelineChanged when calling
ExoPlayer.setShuffleOrder. This in turn resulted in no call to
MediaControllerCompat.Callback.onQueueChanged.
Also make a small fix inside ExoPlayerImpl.setShuffleOrder, to ensure
that the new shuffle order is used when constructing the masked
timeline.
Issue: google/ExoPlayer#9889
#minor-release
PiperOrigin-RevId: 457703727
(cherry picked from commit 5c7ec13e85)
NoUidTimeline still exists as a private detail of TestUtil, but it no
longer extends ForwardingTimeline because the interactions are quite
hard to reason about.
#minor-release
PiperOrigin-RevId: 457703593
(cherry picked from commit 292f6de630)
1. The offloadSchedulingEnabled value doesn't need to be in
PlaybackInfo because it's never updated in EPII.
2. The sleepingForOffload value in EPII wasn't updated explicitly
(just via the return value of a method). It was also only
meant to be enabled while the player is actively playing, but
confusingly triggered from a path where the player may
theoretically be buffering as well.
3. The offload sleeping (=not scheduling doSomeWork) was interwoven
into the actual scheduling code making it slightly hard to follow.
This can be improved slightly by keeping the offload sleeping
decision and the scheduling separate.
PiperOrigin-RevId: 457427293
This will be useful for downgrading to a lower resolution during
a slow preview and for processing slide-shows once sequential
multi-asset editing is supported.
PiperOrigin-RevId: 457017255
videoEncoderFormatUnsupported_completesWithError() has recently
been flaky on API 31 emulators on presubmit because a different
exception than the expected exception is thrown.
This disables it on those emulators to reduce testing noise
until the underlying problem is investigated and resolved.
PiperOrigin-RevId: 456765512
This change is just renaming. There is no functional change intended.
The FrameProcessor interface will be created in a follow-up.
PiperOrigin-RevId: 456741628
After this change GlEffects can use any GlTextureProcessor not just
SingleFrameGlTextureProcessor.
MediaPipeProcessor now implements GlTextureProcessor directly which
allows it to reuse MediaPipe's output texture for its output texture
and avoids an extra copy shader step.
PiperOrigin-RevId: 456530718
After this change, FrameProcessorChain chains any GlTextureProcessors
instead of only SingleFrameGlTextureProcessors.
The GlTextureProcessors are chained in a bidirectional manner using
ChainingGlTextureProcessorListener to feed input and output related
events forward and release events backwards.
PiperOrigin-RevId: 456478414
The offload sleeping stops as soon as a new DO_SOME_WORK message
is handled (because this indicates an expected change in rendering
and we want to stop sleeping until we know it's safe to do so).
Every exit path from doSomeWork needs to clear other pending
DO_SOME_WORK messages as these requests have already been handled by
the current method invocation. This currently doesn't happen from the
offload sleeping return path and a previously queued DO_SOME_WORK
message can immediately wake up the rendering loop again.
Fix this by moving the message removal to the beginning of the
doSomeWork method (as it prevents forgetting it in one of the
exit paths later).
PiperOrigin-RevId: 456259715
In follow-ups the FrameProcessorChain will set an instance of this
listener for each GlTextureProcessor to chain it with its previous
and next GlTextureProcesssor.
PiperOrigin-RevId: 455628942
1. The offloadSchedulingEnabled value doesn't need to be in
PlaybackInfo because it's never updated in EPII.
2. The sleepingForOffload value in EPII wasn't updated explicitly
(just via the return value of a method). It was also only
meant to be enabled while the player is actively playing, but
confusingly triggered from a path where the player may
theoretically be buffering as well.
3. The offload sleeping (=not scheduling doSomeWork) was interwoven
into the actual scheduling code making it slightly hard to follow.
This can be improved slightly by keeping the offload sleeping
decision and the scheduling separate.
PiperOrigin-RevId: 457427293
(cherry picked from commit aedde2de39)
The offload sleeping stops as soon as a new DO_SOME_WORK message
is handled (because this indicates an expected change in rendering
and we want to stop sleeping until we know it's safe to do so).
Every exit path from doSomeWork needs to clear other pending
DO_SOME_WORK messages as these requests have already been handled by
the current method invocation. This currently doesn't happen from the
offload sleeping return path and a previously queued DO_SOME_WORK
message can immediately wake up the rendering loop again.
Fix this by moving the message removal to the beginning of the
doSomeWork method (as it prevents forgetting it in one of the
exit paths later).
PiperOrigin-RevId: 456259715
(cherry picked from commit 251389d744)
Fixing lint errors in the string.xml files makes no sense because these are
overridden with the next automated string import. Adding a lint-baseline.xml
instead for the ui module.
See https://issuetracker.google.com/208178382
#minor-release
PiperOrigin-RevId: 455354304
The wrapper
* catches exceptions for each task and notifies the
listener (this will be used more in follow-ups when processFrame
is split into lots of listeners and callbacks),
* removes finished tasks from the queue and signals any exceptions
that occurred to the listener each time a new task is executed.
PiperOrigin-RevId: 455345184
This change adds a SurfaceProvider interface which is necessary to
allow for texture processors whose output size becomes available
asynchronously in follow-ups.
VTSP's implementation of this interface wraps the encoder and provides
its input surface together with the output frame width, height, and
orientation as used for encoder configuration.
The FrameProcessorChain converts the output frames to the provided
orientation and resolution using a ScaleToFitTransformation and
Presentation replacing EncoderCompatibilityTransformation.
PiperOrigin-RevId: 455112598
The default constructor is only allowed to be called on
API < 32 and the test should use the defined UNSET constant
to be API independent.
#minor-release
PiperOrigin-RevId: 454568893
The native GL errors are in an arbitrary order according to
https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glGetError.xml
This means any of them could contain the most useful message
so it's better to use all for the GlException's message rather than
picking the last message and logging all others.
PiperOrigin-RevId: 454130460
`codecDrainAction` is set to `DRAIN_ACTION_NONE` in 3 places in
`MediaCodecRenderer`:
* The constructor (so there's no prior state to worry about)
* `updateDrmSessionV23()`: Where `mediaCrypto` is reconfigured based
on `sourceDrmSession` and `codecDrmSession` is also updated to
`sourceDrmSession`.
* `resetCodecStateForFlush()`: Where (before this change) the action
is unconditionally set back to `DRAIN_ACTION_NONE` and so any
required updated implied by
`DRAIN_ACTION_FLUSH_AND_UPDATE_DRM_SESSION` is not done.
This change ensures that `flushOrReleaseCodec()` handles
`DRAIN_ACTION_FLUSH_AND_UPDATE_DRM_SESSION` before calling .
This probably also resolves Issue: google/ExoPlayer#10274
#minor-release
PiperOrigin-RevId: 454114428
Based on
https://developer.android.com/reference/android/media/MediaCodec#using-an-output-surface,
frame dropping behaviour depends on the target SDK version.
After this change transformer will only use
MediaFormat#KEY_ALLOW_FRAME_DROP if both the target and system SDK
version are at least 29 and default to its pre 29 behaviour where each
decoder output frame must be processed before a new one is rendered
to prevent frame dropping otherwise.
Also remove deprecated Transformer.Builder constructor without a
context and the context setter.
PiperOrigin-RevId: 453971097
Transformer always enabled glAssertionsEnabled, so there should
be no functional change.
ExoPlayer previously disabled glAssertionsEnabled, so GlUtil logged
GlExceptions instead of throwing them. The GlExceptions are now
caught and logged by the callers so that there should also be no
functional change overall.
This change also replaces EGLSurfaceTexture#GlException with
GlUtil#GlException.
PiperOrigin-RevId: 453963741
The track selector will select multi-channel formats when those can be
spatialized, otherwise the selector will prefer stereo/mono audio
tracks. When the device supports audio spatialization (Android 12L+),
the DefaultTrackSelector will monitor for changes in the platform
Spatializer and trigger a new track selection upon a
Spatializer change event.
Devices with a `television` UI mode are excluded from audio channel
count constraints.
#minor-release
PiperOrigin-RevId: 453957269
This change fixes a bug where the player is frozen with HLS chunkless
preparation because the audio stream wrappers are not marked as master
timestamp sources before preparation.
#minor-release
PiperOrigin-RevId: 453941815
Some Player implementations have no playlist capability but can still
set a MediaItem for playback. Examples are a MediaController connected
to a legacy MediaSession, ExoPlayer up to 2.12 or MediaPlayer.
To indicate this capability, we need an allowed command in addition
to COMMAND_CHANGE_MEDIA_ITEMS that just allows to set a single item
that replaces everything that is currently played.
#minor-release
PiperOrigin-RevId: 453879626
This removes the prior restriction of needing to remember not to crop and set aspect ratio in the same Presentation.Builder, and makes each class a bit more targeted.
This is partially made feasible by the past work to merge consecutive
MatrixTransformations into a single MatrixTransformationFrameProcessor, which
ensures that there's no loss in quality between successive MatrixTransformations.
PiperOrigin-RevId: 453660582
With HLS chunkless preparation, audio formats may have no value
for channel count. In this case, the DefaultAudioSink will either query
the platform for a supported channel count (API 29+) or assume a max
channel count based on the encoding spec in order to decide whether the
audio format can be played with audio passthrough.
Issue: google/ExoPlayer#10204
#minor-release
PiperOrigin-RevId: 453644548
SingleFrameGlTextureProcessor is now an abstract class containing a
default implementation of the more flexible GlTextureProcessor interface
while still exposing the same simple abstract methods for single frame
processing it previously did.
FrameProcessorChain and GlEffect will be changed to use
GlTextureProcessor in follow-ups.
PiperOrigin-RevId: 453633000
The API 32 SDK has incorrect versioning metadata for Spatializer. It
reports the whole class has only been present since API 33 (which is
surely impossible given it's present in the API 32 SDK):
https://issuetracker.google.com/234009300
The metadata seems to be correct in the API 33 SDK, so this baseline
will no longer be needed when we bump to `compileSdkVersion = 33`.
Fixing lint errors in the string.xml files makes no sense because these are
overridden with the next automated string import. Adding a lint-baseline.xml
instead for the ui module.
See https://issuetracker.google.com/208178382
#minor-release
PiperOrigin-RevId: 455354304
(cherry picked from commit 495480a66a)
The default constructor is only allowed to be called on
API < 32 and the test should use the defined UNSET constant
to be API independent.
#minor-release
PiperOrigin-RevId: 454568893
(cherry picked from commit 0ed53215c2)
`codecDrainAction` is set to `DRAIN_ACTION_NONE` in 3 places in
`MediaCodecRenderer`:
* The constructor (so there's no prior state to worry about)
* `updateDrmSessionV23()`: Where `mediaCrypto` is reconfigured based
on `sourceDrmSession` and `codecDrmSession` is also updated to
`sourceDrmSession`.
* `resetCodecStateForFlush()`: Where (before this change) the action
is unconditionally set back to `DRAIN_ACTION_NONE` and so any
required updated implied by
`DRAIN_ACTION_FLUSH_AND_UPDATE_DRM_SESSION` is not done.
This change ensures that `flushOrReleaseCodec()` handles
`DRAIN_ACTION_FLUSH_AND_UPDATE_DRM_SESSION` before calling .
This probably also resolves Issue: google/ExoPlayer#10274
#minor-release
PiperOrigin-RevId: 454114428
(cherry picked from commit c736a72c75)
The track selector will select multi-channel formats when those can be
spatialized, otherwise the selector will prefer stereo/mono audio
tracks. When the device supports audio spatialization (Android 12L+),
the DefaultTrackSelector will monitor for changes in the platform
Spatializer and trigger a new track selection upon a
Spatializer change event.
Devices with a `television` UI mode are excluded from audio channel
count constraints.
#minor-release
PiperOrigin-RevId: 453957269
(cherry picked from commit 9f3c595e02)
This change fixes a bug where the player is frozen with HLS chunkless
preparation because the audio stream wrappers are not marked as master
timestamp sources before preparation.
#minor-release
PiperOrigin-RevId: 453941815
(cherry picked from commit 917484100d)
Some Player implementations have no playlist capability but can still
set a MediaItem for playback. Examples are a MediaController connected
to a legacy MediaSession, ExoPlayer up to 2.12 or MediaPlayer.
To indicate this capability, we need an allowed command in addition
to COMMAND_CHANGE_MEDIA_ITEMS that just allows to set a single item
that replaces everything that is currently played.
#minor-release
PiperOrigin-RevId: 453879626
(cherry picked from commit 2056152824)
With HLS chunkless preparation, audio formats may have no value
for channel count. In this case, the DefaultAudioSink will either query
the platform for a supported channel count (API 29+) or assume a max
channel count based on the encoding spec in order to decide whether the
audio format can be played with audio passthrough.
Issue: google/ExoPlayer#10204
#minor-release
PiperOrigin-RevId: 453644548
(cherry picked from commit b3b57bc93d)
The current setup with distinct, private `keyForField` implementations,
leaves open the (theoretical) possibility of a clash in the `Bundle`
keys used by the superclass and subclass. This change brings
consistency with our only other extensible `Bundleable` type
(`PlaybackException`).
#minor-release
PiperOrigin-RevId: 453385875
(cherry picked from commit a056f08a3d)
We generally nest the `Builder` for `Foo` inside `Foo`. In this case,
there's already a `DefaultTrackSelector.Parameters.Builder` type visible
to a developer, it just happens to be the 'common'
`TrackSelectorParameters.Builder`, so using it is a bit weird. For
example this code snippet doesn't compile because
`DefaultTrackSelector.Parameters.Builder#build()` returns
`TrackSelectionParameters`. This CL fixes that problem and the code
snippet now compiles.
```java
DefaultTrackSelector.Parameters params =
new DefaultTrackSelector.Parameters.Builder(context).build()
```
#minor-release
PiperOrigin-RevId: 453215702
(cherry picked from commit 881622385d)
This is done by removing the `@FieldNumber` IntDef completely. It's not
really adding much value anyway, because it's `open` so there's no real
enforcement to prevent passing 'incorrect' values.
#minor-release
PiperOrigin-RevId: 452108972
(cherry picked from commit 97210b5aa0)
The current setup with distinct, private `keyForField` implementations,
leaves open the (theoretical) possibility of a clash in the `Bundle`
keys used by the superclass and subclass. This change brings
consistency with our only other extensible `Bundleable` type
(`PlaybackException`).
#minor-release
PiperOrigin-RevId: 453385875
We generally nest the `Builder` for `Foo` inside `Foo`. In this case,
there's already a `DefaultTrackSelector.Parameters.Builder` type visible
to a developer, it just happens to be the 'common'
`TrackSelectorParameters.Builder`, so using it is a bit weird. For
example this code snippet doesn't compile because
`DefaultTrackSelector.Parameters.Builder#build()` returns
`TrackSelectionParameters`. This CL fixes that problem and the code
snippet now compiles.
```java
DefaultTrackSelector.Parameters params =
new DefaultTrackSelector.Parameters.Builder(context).build()
```
#minor-release
PiperOrigin-RevId: 453215702
Implementations of this interface will be able to drop or add frames,
change timestamps, accept multiple input frames before producing
output, and process frames on their own background thread.
A default implementation of this interface will be added to SingleFrameGlTextureProcessor in a follow-up.
PiperOrigin-RevId: 453159835
This is done by removing the `@FieldNumber` IntDef completely. It's not
really adding much value anyway, because it's `open` so there's no real
enforcement to prevent passing 'incorrect' values.
#minor-release
PiperOrigin-RevId: 452108972
This internal listener avoids wrapping the TransformationExceptions
in PlaybackExceptions that are handled via the Player.Listener and
is also used for FrameProcessingExceptions which already avoided
the PlaybackException layer previously.
This listener will also be useful in follow-ups for encoder-related
TransformationExceptions that are thrown in the SurfaceProvider that
will be called on the GL thread.
PiperOrigin-RevId: 452074575
This reinstates the permissive behaviour removed by
fe7e5b8181
Test file created by opening bear.opus in a hex editor and naively
duplicating the two header packets, starting at (and including) the
first `OggS` in the file and ending just before the third `OggS`.
#minor-release
Issue: google/ExoPlayer#10038
PiperOrigin-RevId: 452015662
(cherry picked from commit b6b282672c)
This reinstates the permissive behaviour removed by
fe7e5b8181
Test file created by opening bear.opus in a hex editor and naively
duplicating the two header packets, starting at (and including) the
first `OggS` in the file and ending just before the third `OggS`.
#minor-release
Issue: google/ExoPlayer#10038
PiperOrigin-RevId: 452015662
Once the more advanced GlTextureProcessor interface exists,
it will be possible to change the output size of a GlTextureProcessor
between frames. To keep the re-configuration based on the frame sizes
minimal, things indepedent of the frame size, such as the GlProgram,
can be initialized in the constructor.
PiperOrigin-RevId: 451997584
This value only existed to allow setting media URLs from external sources
(e.g. in a MediaController) so that a player can start playing this item.
Now that we have MediaItem.RequestMetadata.mediaUrl we can remove this value
from MediaMetadata because it's request metadata, not media metadata.
PiperOrigin-RevId: 451857413
These fields can be used to transport additional request properties
when the requester doesn't know the details needed for the actual
playback (i.e. the LocalConfiguration).
PiperOrigin-RevId: 451857093
ExoPlayer applies a large time offset to buffers so that, if the input has negative timestamps, generally buffers seen by the decoders should have positive timestamps. Modify how the offset is handled in `Transformer` so that decoders and encoders generally see positive timestamps, by leaving the offset on samples when reading them in the base renderer (remove the code that removed the offset), and then removing the offset when muxing. Also update the frame processor chain and slow motion flattening code to retain the existing behavior after this change (these both need original media presentation timestamps)
Tested via existing end-to-end tests and manually verified that the overlay frame processor shows the expected original media timestamps.
Aside: we don't need the same logic as ExoPlayer to track stream offsets across the decoder yet, because we don't try to handle stream changes during playback in single asset editing. (There is an edge case of multi-period DASH that may not work but I doubt anyone will use that as input to `Transformer` before we change the code to handle multi-asset properly.) In future we should try to refactor interaction with the decoder to use the same code for Transformer and ExoPlayer.
PiperOrigin-RevId: 451846055
Most devices won't support 8k decoding, so they'll skip this test entirely.
As the video is quite short, this test shouldn't be any longer than the nearby,
long-running 4k60 test.
PiperOrigin-RevId: 451423368
Unconditionally sleep for offload, if the audio buffer is full.
Previously ExoPlayer would not sleep if the expected wake-up was
in 2s. This was to prevent underrun if the wake-up was delayed.
Experiments have shown that the wakup audio buffer is far more
than 2s (around 1min). Additionally,
the metric was incorrect because it measured both,
AudioTrack + DSP.
Finally, this metric was erroneous after a gapless transition,
when the head position would reset to 0 and thus the computed
delay until next wakeup was too large.
PiperOrigin-RevId: 451383701
SSIM calculation requires the input and output dimensions to be identical.
For devices that can't encode the input dimensions, skip SSIM calculations and
log the cause. Only apply this on tests where the encoder may not support the
input file dimensions.
PiperOrigin-RevId: 451364904
Decode-only video frames (needed when the frame at / first frame after the
clipping start is not a key frame) need to be decoded but not passed to
the frame processor chain or encoder.
The clipping start offset needs to be removed from the frame timestamps
in the passthrough and video pipelines.
There are no changes needed for this in the audio pipeline, as it doesn't
use the input timestamps -- it uses its own timestamps derived from the
buffer sizes instead.
Also add demo option to try this out.
#minor-release
PiperOrigin-RevId: 451353609
Also update names of implementations to match design doc.
In follow-ups, SingleFrameGlTextureProcessor will become
an abstract implementation of a new GlTextureProcessor
interface.
Texture processor makes sense as it processes OpenGL textures.
The term frame processor will be used for something else in
follow-ups.
PiperOrigin-RevId: 451142085
* Rename (via deprecation)
`MediaItem.DrmConfiguration.Builder#forceSessionsForAudioAndVideoTracks`
to `setForceSessionsForAudioAndVideoTracks`. This is more consistent
with existing 'force' method names both in this class and in
`TrackSelectionParameters.Builder`.
* Add missing `@Nullable` annotation to the parameter for
`MediaItem.SubtitleConfiguration.Builder#setMimeType`. This annotation
is already present on the `MediaItem.SubtitleConfiguration#mimeType`
field that this setter corresponds to.
PiperOrigin-RevId: 450941336
There is a problem with the ImageReader formats used by the
SSIM helper that only occurs for Nexus 5 API 21, so as a workaround
we can skip the SSIM calculation on Nexus 5 API 21.
This skips just the SSIM calculation (by setting the value to
1.0 instead and logging). The tests still run when SSIM is skipped
so that we can detect other failures.
PiperOrigin-RevId: 450903183
This value is only needed by subclasses of `Rating`, all of which are
in this package (the `Rating` constructor is already package-private to
ensure this).
PiperOrigin-RevId: 450886872
*** Original commit ***
Rollback of 57182ac7bd
*** Original commit ***
Remove `@Nullable` from `MediaSource.Factory` setters
The null-behaviour of these methods creates a minimization footgun,
because **any** call to these setters will prevent R8 from removing
the default implementation (even if it's never used by the app) - this
is because R8 can't tell the default imple...
***
PiperOrigin-RevId: 450453325
AudioTrack.setOffloadEndOfStream should be called after a track
has been buffered. Additionally, the AudioTrack must be playing.
It has been observed that for very short media (<1s), the AudioTrack
might not have started immediately after the read that buffered
the audio.
In such a situation, calling AudioTrack.setOffloadEndOfStream throws
and playback fails.
Avoid this failure by checking that the AudioTrack is playing before
calling setOffloadEndOfStream.
This means that very short gapless media will not be gapless, this was
deemed acceptable given that such very short media should be very rare
in offload.
PiperOrigin-RevId: 450431146
*** Original commit ***
Remove `@Nullable` from `MediaSource.Factory` setters
The null-behaviour of these methods creates a minimization footgun,
because **any** call to these setters will prevent R8 from removing
the default implementation (even if it's never used by the app) - this
is because R8 can't tell the default implementation is only used if the
parameter is `null`.
******
PiperOrigin-RevId: 450410833
The null-behaviour of these methods creates a minimization footgun,
because **any** call to these setters will prevent R8 from removing
the default implementation (even if it's never used by the app) - this
is because R8 can't tell the default implementation is only used if the
parameter is `null`.
Follow-up to 57182ac7bd
PiperOrigin-RevId: 450395941
The null-behaviour of these methods creates a minimization footgun,
because **any** call to these setters will prevent R8 from removing
the default implementation (even if it's never used by the app) - this
is because R8 can't tell the default implementation is only used if the
parameter is `null`.
PiperOrigin-RevId: 450386627
This detection relies on an unsupported workaround and may trigger
permission warnings in tools analyzing permission usage although
no permission is needed or requested by app code.
Given the majority of 5G-NSA playbacks are on API 31+ by now,
we can remove this path to avoid the permission confusion and the
unsupported detection workaround.
PiperOrigin-RevId: 450382586
The debug surface view's output surface can become invalid during a transformation due to the parent activity pausing, for example. This can currently cause a crash when backing out of the `TransformerActivity` in the demo because the surface can be destroyed before the transformer has fully canceled.
Also clarify naming of the outputSurface and inline the private method that created `EGLSurface`s (it was shorter after removing the debug preview).
PiperOrigin-RevId: 449963440
This causes a bug where the forwarded selections are no longer
assumed equal and the child MediaPeriods will think they need
to reset streams even though the selection stayed the same.
Issue: Issue: google/ExoPlayer#10248
PiperOrigin-RevId: 449454038
We need to pass timestamp for the list of cues so we are defining a new class CueGroup which will store both cues and timestamp.
PiperOrigin-RevId: 449212054
The old reference was just for prototyping HDR. The new reference is for planned work to use the correct formats for input and output for HDR editing in GL.
PiperOrigin-RevId: 449211792
The MMWAVE constant was deprecated in favour of a new constant
with a better name. Thus, we need to check for both constants now.
PiperOrigin-RevId: 449018959
This is an individual language (ISO 639-3) part of the Arabic
macrolanguage ("ar" in ISO 639-1). Add this mapping to our
existing list similar to other individual to macrolanguage
mappings we have already.
Issue: Issue: google/ExoPlayer#10255
PiperOrigin-RevId: 448911950
When using a MatrixTransformationFrameProcessor per transformation
matrix, each frame processor's shader applies the matrix to the
vertices and clips the result to the NDC range when drawing the
output frame.
This change combines consecutive MatrixTransformations into a single
MatrixTransformationFrameProcessor by multiplying the individual
matrices while updating and clipping the visible polygon after
each matrix and mapping the resulting visible polygon back to the
input space so that its vertices and the combined transformation
matrix can be used in the shader.
PiperOrigin-RevId: 448521068
Network type detection on these API levels couldn't be tested
yet because of a missing Robolectric feature. This was fixed by
the recent Robolectric upgrade and the restrictions can be removed.
This also requires to replicate the platform hack we rely on on
these API levels.
PiperOrigin-RevId: 448240431
Some RTP foramts are statically assigned, so they don't have the rtpmap
attribute. Create the missing rtpmap attribute in this case.
PiperOrigin-RevId: 448239724
This listener replaces
FrameProcessorChain#getAndRethrowBackgroundExceptions.
The listener uses a new exception type FrameProcessingException
separate from TransformationException as the frame processing
components will be made reusable outside of transformer soon.
PiperOrigin-RevId: 447455746
This constant is used for https://docs.gl/es2/glVertexAttribPointer
which takes the number of components per generic vertex attribute
(meaning the size of the individual coordinate vectors here) not the
number of attributes (the number of vertices that the old constant
name referred to).
PiperOrigin-RevId: 447427241
Before this change, we list the formats for which we don't
want transcoding. This change disables transcoding altogether.
This was tested by checking that transcoding takes place on a
short camera recording only when the added flag is not present
(and AndroidManifest does not declare support for HEVC).
PiperOrigin-RevId: 446986580
We won't try to provide/rethrow helpful error messages that are already
provided by GL, as this sort of task would expand into writing a GL verifier.
A Gl verifier is unnecessarily complex for minimal payoff, especially as Apps
expected to read GL error messages would mostly be those writing custom
GlFrameProcessors, who should be already be familiar with reading GL error
messages anyways.
PiperOrigin-RevId: 446950837
ScaleToFitFrameProcessor, PresentationFrameProcessor,
and EncoderCompatibilityFrameProcessor now each implement
MatrixTransformation instead of wrapping
MatrixTransformationFrameProcessor.
PiperOrigin-RevId: 446480286
The TrackSelector is released when the player is released. The
TrackSelector can be reused if TrackSelector.init() is called again.
PiperOrigin-RevId: 446439717
This change splits AdvancedFrameProcessor into 4 files:
- MatrixTransformationFrameProcessor for the GlFrameProcessor
implementation
- MatrixTransformation and GlMatrixTransformation for the GlEffect
specification
- MatrixUtils for the static matrix helpers
PiperOrigin-RevId: 446236384
Some APIs from Android 12L were used either via reflection or
constants values were hard-coded. We can now use these APIs directly
since we upgraded the compile SDK version to 32.
PiperOrigin-RevId: 446167543
Also add warning for what can happen if this is ignored.
Previously the return value was overridden by setOutputSurface, now
setOutputSurface only overrides the values using internally.
PiperOrigin-RevId: 445377036
Developers are expected to (eventually) only use methods on
PlayerView (and not PlayerControlView) to interact with the UI
controller.
PiperOrigin-RevId: 445361488
This is consistent with the IntDef name, and frees up the CONTENT_TYPE_
prefix for the @ContentType values (which are currently just TYPE_*,
and therefore ambiguous with lots of other 'type' values in C).
PiperOrigin-RevId: 445356476
This method was introduced in 754eb1527a
as a replacement for Util.inferContentType(String) but it incorrectly
didn't return TYPE_SS when passed "ism" or "isml".
PiperOrigin-RevId: 445217167
We add an entire class like we do for parsing other codec initialization formats; it's currently not doing any parsing though (... initialization data is really simple for AV1 though: just the entire contents of the box).
For testing, we add the sample file, having been re-encoded with ffmpeg (and we also happen to have another av1 file, too).
PiperOrigin-RevId: 444890282
This fixes some small niggles:
1. `inferContentType(String)` is documented to take a path, but in the
tests we're passing full URIs.
2. A `String` parameter is usually a path, but also a MIME type or an
extension. In the new methods, the meaning of a `String` parameter
is always clear from the name of the method.
3. `inferContentType(String)` is always passed an extension in
'production' code (which has to be manually prefixed with a dot).
4. `inferContentType(Uri, @Nullable String)` always ignores the Uri if
the String is non-null. IMO this logic is clearer to a reader if it's
just in-lined at the call-site.
These methods are used from the demo apps, so will be part of the stable
API.
PiperOrigin-RevId: 444826053
To ensure frame processor operations operate on square pixels,
make the frame taller or wider for non-square input pixels.
In addition to automated tests, this was tested by changing the
inputFormat.pixelWidthHeightRatio in the TransformerVideoRenderer.
PiperOrigin-RevId: 444553517
This allows the actual bitmap to be saved, even if output
dimensions are different than expected. Otherwise, differing
output dimensions would throw an exception, preventing the bitmap
from being saved.
PiperOrigin-RevId: 444512210
The last frame processor could use a different drawing command
than a four vertex triangle strip. So we need call its drawing
method again instead of assuming the shape.
PiperOrigin-RevId: 444217274
Avoid no-op FrameProcessors in VideoTranscodingSamplePipeline, to avoid
creating resources for and executing GL for a no-op vertex+fragment shader.
EncoderCompatibilityFrameProcessor will still always be exercised.
PiperOrigin-RevId: 443675833
This test should run on all devices from API 21 (the media uses Baseline
profile level 3.0 H.264) to give us coverage of the full pipeline (forcing
re-encoding) and SSIM calculation on all devices.
PiperOrigin-RevId: 443650002
Split rotationDegrees changes to EncoderCompatibilityFrameProcessor, a new
FrameProcessor.
This removes automatic rotationDegrees adjustments from Presentation, which
allows Presentation to be used for changes before the end of a
FrameProcessorChain pipeline.
PiperOrigin-RevId: 443387226
The issue with decoding getting stuck is still reproducible on `samsung/beyond1lteeea/beyond1:12/SP1A.210812.016/G973FXXUEGULB:user/release-keys`.
PiperOrigin-RevId: 443368546
Some devices under report their resolution support, like 2144 for 2160 in
H265, 1072 for 1080 in H264. This CL only takes care of these two cases,
- reporting 1920x1080 is supported when the device reports 1920x1072, and
- reporting 3840x2160 is supported when the device reports 3840x2144
PiperOrigin-RevId: 443095042
When downlading an adaptive asset, if an ExoPlaybackException happens
during track selection, the player raises an
UnsupportedOperationException which is not handled gracefully and can
crash the app main thread.
This change catches the error and forwards it to
DownloadHelper.Callback.onPrepareError() as an IOException.
PiperOrigin-RevId: 443015332
This is safer because it will prevent any future problems with
creating a new FrameProcessorChain before the previous one has
completed its async release.
From [eglDestroyContext documentation](https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglDestroyContext.xhtml):
"If the EGL rendering context is not current to any thread,
eglDestroyContext destroys it immediately."
The context isn't current to any thread here because GlUtil calls
eglMakeCurrent with EGL_NO_CONTEXT before calling eglDestroyContext.
So everthing should be released once the FrameProcessorChain's
release task terminates.
PiperOrigin-RevId: 442807484
* Group what's now many related test PNGs by moving them to their own directory.
* Move bitmap references to files where they're used, as each bitmap is only
used once each, except the original bitmap.
PiperOrigin-RevId: 441485489
Scaling and rotation using ScaleToFitFrameProcessor may change the
the resolution and aspect ratio, so defaulting to the same as the
input is no longer accurate.
PiperOrigin-RevId: 441463349
What a minimal implementation should include is now explained in the
interface javadoc while the method name reflects what the method does.
PiperOrigin-RevId: 441432059
The stream with the mp3 track is added because
- We only encode to AAC
- We only encode when the source track is not AAC
Now that we have a way to force encoding, we no longer need the mp3 track.
The test asset is kept for later parameterized testing.
PiperOrigin-RevId: 440876080
The MIME type was set to H265 to force transcoding. Now that we have an encoder
factory that forces encoding, switching back to H264 ensures the quality test
is conducted on more devices (those don't support H265 can be tested now).
However, H265 should be part of the quality test after we have proper mechanism
to skip test based on device capability.
PiperOrigin-RevId: 440132471
The DefaultMediaNotificationProvider checks if a command is available
before putting the respective action in the notification.
PiperOrigin-RevId: 440114422
We don't currently have enough understanding of the correlation between a
specific SSIM score and video quality. Dropping to .90 to make most tests pass.
Especially when there's no discernible difference from the videos with .9 and
.95 SSIM.
PiperOrigin-RevId: 440047551
PeriodicDimmingFrameProcessor is an example of how a custom fragment
shader can be used to apply color changes that change over time.
PiperOrigin-RevId: 439840609
The matrix provider allows the transformation matrix to be updated
for each frame based on the timestamp.
The following example effects using this were added to the demo:
* a zoom-in transition for the start of the video,
* cropping a rotating rectangular frame portion,
* rotating the frame around the y-axis in 3D.
PiperOrigin-RevId: 439791592
We add an entire class like we do for parsing other codec initialization formats; it's currently not doing any parsing though (... initialization data is really simple for AV1 though: just the entire contents of the box).
For testing, we add the sample file, having been re-encoded with ffmpeg (and we also happen to have another av1 file, too).
PiperOrigin-RevId: 439453823
This allows apps to use AdvancedFrameProcessor to apply transformations
in 3D space. This functionality is not used in transformer otherwise.
PiperOrigin-RevId: 439313406
The encoder surface is no longer needed for the OpenGL setup and frame
processor initialization, as a placeholder surface is used instead. So
all of the setup can now be done in the factory method.
PiperOrigin-RevId: 438844450
App code should get all of this information from TrackGroupInfo,
and should only need TrackGroup as a key to use for overrides.
PiperOrigin-RevId: 438840925
This allows the MuxerWrapper to keep using trackTypeToTimeUs for
calculating the video duration but slightly changes the meaning of
its interleaving constraints.
PiperOrigin-RevId: 438780686
This provides better compatibility with MediaExtractor, which does read these fields; we also need them for being able to mux file contents into another mp4 file.
Also, there is a minor refactor included so that we have an actual type for esds box contents instead of a pair.
PiperOrigin-RevId: 438673825
The placeholder surface is either EGL_NO_SURFACE or a 1x1 pbuffer
depending on whether the device supports EGL_KHR_surfaceless_context.
PiperOrigin-RevId: 438541846
Since the output textures and surfaces are managed by the
FrameProcessorChain, clearing them there makes sense.
This is also less error-prone as it might not be obvious to
someone implementing a GlFrameProcessor that they need to
glClear. (Clearing twice won't cause any problems.)
PiperOrigin-RevId: 438532247
This requires an additional nanos to micros conversion because
the SurfaceTexture uses nanos. But as the timestamps from the
MediaCodec decoder (propagated in DefaultCodec#releaseOutputBuffer) are
in microseconds no precision is lost here.
Also add test that checks output video duration.
PiperOrigin-RevId: 438010490
MAXIMUM_AVERAGE_PIXEL_ABSOLUTE_DIFFERENCE was copied from a test
class, but BitmapTestUtil isn't a test. So the javadoc needs
rewording to reflect that.
PiperOrigin-RevId: 438001833
The problem is not the IntDef array, it's the fact the lint tool
is unable to correctly infer the annotations on the lambda parameters
without them being explicitly annotated. It seems explicitly annotating
is better than suppressing all IntDef warnings in the whole method.
PiperOrigin-RevId: 437969271
From Android T onwards `MediaCodec` supports requesting tone-mapping down to
SDR. Add an option to request this behavior and document that it isn't
supported before T. Also add an option in the demo app to try it out.
Tested manually on a prerelease build.
PiperOrigin-RevId: 437765325
The thread name is used to verify the thread in both createOpenGlObjectsAndInitializeFrameProcessors() and processFrame().
Also remove glThread field that was only used for this verification.
PiperOrigin-RevId: 437730804
Since the output size can be overridden, the viewport should be
ouputWidth/Height and NOT the ExternalCopyFrameProcessor's output size
which matches the input size.
PiperOrigin-RevId: 437256635
OutlineColour should be treated as the background color if BorderStyle=3. Since currently BorderStyle is ignored, we can always treat OutlineColor as the background color.
* The AdvancedFrameProcessor calls use() in updateProgramAndDraw().
* The AdvancedFrameProcessor has the same input and output dimensions.
PiperOrigin-RevId: 437231350
On some devices, decoding gets stuck when the number of frames pending at the
`SurfaceTexture` is too high. We added a workaround that only allows one frame
to be pending at a time. That fixed the issue, however, based on on-device
testing it seems that it's safe to queue more than one frame.
Add a method that returns a safe estimate of the number of frames that can be
pending at a time, and use this to limit the number of frames that can be
released from the decoder but not processed by the frame processor chain.
PiperOrigin-RevId: 437057075
Configuring the frame sizes between frame processors is now the
FrameProcessorChain's rather than the caller's responsibility.
The caller can getOutputSize() and override it for encoder fallback
in configure().
PiperOrigin-RevId: 437048436
The factory method is replaced by a public constructor and
configure() method which configures the input/output surfaces
and handles the OpenGL setup.
This is a prerequisite for removing the responsibility of the
caller to configureSizes() before creating the chain in a follow-up.
PiperOrigin-RevId: 437028882
This allows us to bypass many device-specific issues, that only occur when
decoding directly to an encoder surface, without OpenGL. This also allows us
to maintain fewer code branches, which require additional testing to verify
correctness.
PiperOrigin-RevId: 437003138
Allow apps to modify how frames are presented, via modifying resolution.
A follow-up CL will provide aspect ratio, cropping, etc.
PiperOrigin-RevId: 436963312
* These grouping parentheses are unnecessary; it is unlikely the code will be misinterpreted without them
(see http://go/bugpattern/UnnecessaryParentheses) (11 times)
This CL looks good? Just LGTM and Approve it!
This CL doesn’t look good? This is what you can do:
* Revert this CL, by replying "REVERT: <provide reason>"
* File a bug under go/error-prone-bug for category ErrorProneStyle if there's an issue with the CL content.
* File a bug under go/rosie-bug if there's an issue with how the CL was managed.
* Revert this CL and not get a CL that cleans up these paths in the future by
replying "BLOCKLIST: <provide reason>". This is not reversible! We recommend to
opt out the respective paths in your CL Robot configuration instead:
go/clrobot-opt-out.
This CL was generated by CL Robot - a tool that cleans up code findings
(go/clrobot). The affected code paths have been enabled for CL Robot in //depot/google3/java/com/google/android/libraries/media/METADATA which is reachable following include_presubmits from //depot/google3/third_party/java_src/android_libs/media/METADATA.
Anything wrong with the signup? File a bug at go/clrobot-bug.
#codehealth
Tested:
Local presubmit tests passed.
PiperOrigin-RevId: 436726763
SEF is similar to frame editing, where the input and output videos are
not intended to match, so SSIM will not provide a reliable value. To
check SSIM correctly in future, we would need to provide
golden/expected video files to compare output against.
PiperOrigin-RevId: 436707240
The FrameProcessorChain manages a List<GlFrameProcessor>.
FrameProcessorChainDataProcessingTest now tests chaining ScaleToFit-
and AdvancedFrameProcessors.
PiperOrigin-RevId: 436468037
This currently causes the test to fail on Pixel 6 Pro running a recent S build
SQ1D.220205.004.
There is no need to test audio transcoding while we are measuring video
quality.
PiperOrigin-RevId: 435635314
ExternalCopyFrameProcessor's output dimensions match the input
size not the output size. So the intermediate texture size
should match the input size.
Also rename configureOutputDimensions to configureOutputSize.
PiperOrigin-RevId: 435058789
More info: go/lsc-assertthrows and go/assertthrows
NOTE: if the source of truth for this code is _NOT_ `//third_party/`, please ask for this CL to be reverted.
Tested:
TAP --sample ran all affected tests and none failed
http://test/OCL:434925976:BASE:434869111:1647399186064:de338189
PiperOrigin-RevId: 435047509
Some infra thinks the if does not protect against API incompatibilities
(example: Android's soong build system). AndroidStudio 2021.3.1 also
signals a warning.
#minor-release
PiperOrigin-RevId: 435027073
Use android.util.Size, whose naming is much easier to understand than Pair<Integer, Integer>, in both FrameProcessor and EncoderUtil.
PiperOrigin-RevId: 434813986
* Move auto-adjustments for transformation matrices from the
VideoTranscodingSamplePipeline constructor to the new
ScaleToFitFrameProcessor.
* Add GlFrameProcessor#getOutputDimensions() to allow for GlFrameProcessors with
different input and output dimensions. This is a prerequisite for
Presentation.
* Tested with unit tests (and manually just in case).
* A follow up CL will implement change the FrameProcessor input to be scale and
rotate values as requested by the user. This was kept out of this CL to
reduce CL review size. Presentation will also be implemented in a follow up
CL.
PiperOrigin-RevId: 434774854
As pointed out in a previous review, createFallbackTransformationRequest
can be a bit confusing to parse.
Added javadocs and renamed parameters appropriately, to make it slightly
more easy to understand.
PiperOrigin-RevId: 434733313
*** Original commit ***
Rollback of 20282151b5
*** Original commit ***
Fix PlayerView touch handling
Overriding onTouchEvent was causing multiple issues, and
appears to be unnecessary. Removing the override fixes:
1. StyledPlayerView accessibility issue where "hide player
controls" actually toggled play/pause.
2. Delivery of events to a registered OnClick...
***
PiperOrigin-RevId: 434502423
*** Original commit ***
Don't call MediaDrm.setLogSessionId in FrameworkMediaDrm
This method throws an UnsupportedOperationException on some Android 12
devices.
***
PiperOrigin-RevId: 433708582
The variable marks the coordinates used to sample from a texture, so hopefully
this makes the naming a bit more descriptive.
This renames vTexCoords and aTexCoords. No functional changes intended.
PiperOrigin-RevId: 433499934
This fixes an exception thrown when parceling becasue the type can not be found
(expects the AdsPlaybackState to be Serializable). Transforming the map and the
ad playback states to a Bundle fixes the problem.
#minor-release
PiperOrigin-RevId: 433491993
When the start position of a MediaPeriodInfo is equal or higher than the duration,
we set the start position to `duration - 1` to end on the last frame. With server
side inserted ad streams, this has the effect that we actually need to seek back to
the last content frame after a post-roll.
This is desirable when actually ending on that frame but produces a BUFFERING event
when transitioning from an SSAI stream with a post-roll to the next media item in
the playlist. This change sets the start position to the duration when we are
clipping the last content period of an SSAI stream that is played in a playlist.
PiperOrigin-RevId: 433445680
If an OpenGL call blocks because the encoder's input surface is full,
this will now block the background thread while the main thread can
continue querying encoder output and free up encoder capacity until
it accepts more input unblocking the background thread.
PiperOrigin-RevId: 433283287
*** Original commit ***
Fix PlayerView touch handling
Overriding onTouchEvent was causing multiple issues, and
appears to be unnecessary. Removing the override fixes:
1. StyledPlayerView accessibility issue where "hide player
controls" actually toggled play/pause.
2. Delivery of events to a registered OnClickListener when
useController is false.
3. Delivery of events to a registered OnLongClickListener
in all configurations.
4. Incorrectly treating a sequence of touch events that
exit the bounds of the vi...
***
PiperOrigin-RevId: 433262414
With the new version, we try the following before fixing resolution:
- Fix size alignment
- Try 3/4 the width and height
- Try 2/3 the width and height
- Try 1/2 the width and height
Also: align the resolution ends in 1 or 9 to 0.
PiperOrigin-RevId: 433206358
Overriding onTouchEvent was causing multiple issues, and
appears to be unnecessary. Removing the override fixes:
1. StyledPlayerView accessibility issue where "hide player
controls" actually toggled play/pause.
2. Delivery of events to a registered OnClickListener when
useController is false.
3. Delivery of events to a registered OnLongClickListener
in all configurations.
4. Incorrectly treating a sequence of touch events that
exit the bounds of the view before ACTION_UP as a click,
both for delivery to OnClickListener and for toggling
the controls.
Note: After this change, control visibility will not be
toggled if the application developer explicitly sets the
view to be non-clickable. I think that's probably working
as intended though. It seems correct that a non-clickable
view would not respond to clicks.
Issue: google/ExoPlayer#8627
Issue: google/ExoPlayer#9605
Issue: google/ExoPlayer#9861
PiperOrigin-RevId: 433016626
This change rewrites the UI module's track selection
components to depend on the Player API, allowing us to
finally remove the UI module's dependency on ExoPlayer
as a concrete player implementation.
PiperOrigin-RevId: 432989318
Previously, we've used getSupportedHeights/Widths() to find the supported
resolution. However, the height/width can be over-reported when using these
APIs. For example, getSupportedWidths and getSupportedHeights can both return
3840, but the supported height when using 3840 as width is only 2160.
PiperOrigin-RevId: 432926192
This makes the reading period advance early as expected at the end of an ad
period. Before this change the reading position of the metadata renderer
prevented advancing the period until metadata arrived after the start position of
the following period. Only then the reading position of the metadata renderer
is updated and beyond the start position of the following period which is a
condition to advance the reading period.
Because transitioning to the next period is a virtual transition and the
SharedMediaPeriod keeps reading from the same underlying sample streams, the
metadata renderer can safely be ignored for this check.
#minor-release
PiperOrigin-RevId: 432646037
We will be migrating our track selection UI components to be
based on TracksInfo. We need DownloadHelper to expose TracksInfo
to make it compatible with such components.
PiperOrigin-RevId: 432474487
addTrackSelectionForSingleRenderer takes a list of legacy overrides,
which are then set on the supplied parameters one at a time to run
track selection. This allows multiple overrides for a single track
type to be applied in the download use case, despite it not being
possible to place such overrides directly into a single parameters.
For new style overrides, multiple overrides for the same track type
can be placed directly into a single parameters. Therefore we'll be
able to replace use of addTrackSelectionForSingleRenderer with use
of addTrackSelection, which is a much cleaner API. For this to work,
we need to make DownloadHelper apply multiple overrides in this case.
PiperOrigin-RevId: 432459834
This will allow for easier customisation of the additional tasks
performed by the test runner, such as calculating metrics like SSIM.
PiperOrigin-RevId: 432434850
These should have been removed as part of 1391b7c65d, since we no
longer officially support overriding the layout file for this class.
This class is known as StyledPlayerView in exoplayer2.
#minor-release
PiperOrigin-RevId: 432411322
This method is no longer needed since we added SubtitleConfiguration#id
in 59d98b9a4e.
Issue: google/ExoPlayer#10016
#minor-release
PiperOrigin-RevId: 432169262
DownloadHelper is in the ExoPlayer module, so there's no reason
why it can't use ExoPlayer specific track selections. That said,
we want our UI components to operate on generic
TrackSelectionParameters, and we want such UI components to be
useful for selecting tracks for download. To keep this interop,
it's necessary to have DownloadHelper accept generic
TrackSelectionParameters, or to require application code to
convert them. The first approach seems preferable!
PiperOrigin-RevId: 432158846
This constructor always does the wrong thing for non-adaptive groups
containing more than 1 track, because it'll incorrectly generate an
adaptive selection. Replace it with a constructor for specifying a
single track within the group instead.
PiperOrigin-RevId: 431673458
Only log GL exceptions in GlUtil if they aren't thrown. Otherwise,
it's up to the caller whether or not to log them to avoid logging them
twice.
PiperOrigin-RevId: 431657397
This test tests the same cases as the FrameEditorDataProcessingTest
as currently the main FrameEditor functionality is to apply a
transformation matrix using a TransformationFrameProcessor.
PiperOrigin-RevId: 431642066
As evidenced by the somewhat awkward logic in PlayerControlView, the
previous design wasn't very friendly to expected usage. There will be
more usage when the track selection dialog components are migrated,
which would be similarly awkward without this change.
PiperOrigin-RevId: 431407675
We use SSIM to measure the transcoding quality between. SSIM is a widely used
tool that compares the luma channel between two images, and generates a score
from 0 to 1 that indicates "how similar" the two images are.
In `SsimHelper`, we decode the two videos, extract matching frames and
calculates the mean SSIM (SSIM averaged all matching frames) for both videos.
Matching frames are referred to as "comparisonFrame" in the CL, which is
selected based on the frame number and a user-set comparison interval.
For instance, if the interval is 7, then every seventh frames are compared.
We use MediaCodec/MediaExtractor to decode the video, and use ImageReader to
extract the decoded frame.
The SSIM calculation logic is a inspired by and modified from the CTS
[MSSIMMatcher](https://cs.android.com/android/platform/superproject/+/master:cts/tests/tests/uirendering/src/android/uirendering/cts/bitmapcomparers/MSSIMComparer.java;l=1?q=mssimcom)
that has some errors and extra features we don't need (like handling RGB
images).
Adds TranscodeQualityTest to ensure high quality transcoding.
PiperOrigin-RevId: 430951206
*** Original commit ***
Rollback of 8d9c4f4774
*** Original commit ***
Rollback of 43b796b64d
*** Original commit ***
Rollback of 08c5b1cb0a
*** Original commit ***
PiperOrigin-RevId: 430905772
The AsyncronousMediaCodecAdapter should call MediaCodec.start()
on the same thread it calls MediaCodec.flush(), i.e. the playback
thread. This change removes the experimental flag that allowed
calling MediaCodec.start() from the callback thread.
The flag was flipped to true already.
PiperOrigin-RevId: 430689665
This can fix the following warning caused by commit a7aa84a:
Warning: com.google.android.exoplayer2.drm.UnsupportedDrmException.Reason:
The typedef annotation should have @Retention(RetentionPolicy.SOURCE)
Author: Tony Guo <tony.guo.peng@gmail.com>
Date: Mon Feb 28 10:56:52 2022 +0800
