Commit graph

560 commits

Author SHA1 Message Date
leonwind
d15f2ed388 Implement RGBA color matrices processor
* Transform frame colors using a defined RGBA Matrix to apply filters.

PiperOrigin-RevId: 464523581
2022-08-08 08:36:03 +00:00
leonwind
ae1915ea1a Migrate Contrast asset to emulator generated one.
PiperOrigin-RevId: 464519030
2022-08-08 08:35:12 +00:00
huangdarwin
687a50e9b3 HDR: Have @C.ColorTransfer check consider NO_VALUE as SDR.
Format.NO_VALUE is a placeholder value for an invalid @C.ColorTransfer, used
for example when the decoder doesn't support this transfer function.

When encountering this invalid value, interpret this as COLOR_TRANSFER_SDR.

Confirmed locally that an exception is thrown when transcoding on p4head, and no exception is thrown when transcoding with this CL.

PiperOrigin-RevId: 464135080
2022-08-08 08:34:12 +00:00
hschlueter
480c129c54 Move FrameProcessor and related interfaces to common.
This will allow effects preview in ExoPlayer to use the
Effect and FrameProcessor interface (and the interfaces
they depend on) without depending on transformer or the
future effects module.

PiperOrigin-RevId: 464060047
2022-08-08 08:33:22 +00:00
huangdarwin
534740fd46 Transformer: Improve misc javadocs and exception logs.
PiperOrigin-RevId: 463971447
2022-08-08 08:26:35 +00:00
huangdarwin
65a2a3a0a0 HDR: Centralize getting Colorinfo from MediaFormat
* Sets KEY_HDR_STATIC_INFO from MediaFormat in the DefaultCodec.
* Adds checks in mediaparser to ensure color space, range, and transfer are valid
  values.

PiperOrigin-RevId: 463921325
2022-08-08 08:24:51 +00:00
hschlueter
0f96d8615d Generalize frame processing error codes.
TransformationException error codes were previously tied to OpenGL
but other FrameProcessor implementations are possible. So this CL
renames the error codes.
Also, remove GL_INIT_FAILED error code, as FrameProcessor
implemenations may initialize resources on a background thread
after the factory method returns, so it's not obvious how to
distinguish between initialization failures and processing failures.

PiperOrigin-RevId: 463704902
2022-08-08 08:17:46 +00:00
hschlueter
580e44fc47 Add Effect marker interface.
This allows non-GL effects to be passed to custom FrameProcessor
implementations.

PiperOrigin-RevId: 463696384
2022-08-08 08:16:50 +00:00
hschlueter
b28fb13998 Remove redundant interface from FinalMatrixTPWrapper.
PiperOrigin-RevId: 463646211
2022-08-08 08:15:55 +00:00
hschlueter
b1b9ba825f Make FrameInfo and SurfaceInfo public.
Both are used in the public FrameProcessor interface, so they
should be public too.

PiperOrigin-RevId: 463454859
2022-08-08 08:14:59 +00:00
samrobinson
56e50b63ec Add 8 new videos, representing lower resolutions.
Downsampled from 3840x2160 resolution videos using ffmpeg:

`ffmpeg -i <inputFilename> -filter:v scale=<width>:<height> -c:a copy <outputFilename>`

PiperOrigin-RevId: 463454628
2022-08-08 08:14:03 +00:00
samrobinson
60d6654b8d Add H265 as a parameter to SsimMapperTest.
PiperOrigin-RevId: 463452893
2022-08-08 08:13:05 +00:00
leonwind
302a742c94 Rename ContrastProcessorPixelTest and make contexts final
PiperOrigin-RevId: 463450395
2022-08-08 08:12:15 +00:00
samrobinson
4a0b07b4f7 Add an SsimMapperTest, which binary searches for 95% ssim.
This is possible because SSIM increases monotonically with bitrate.

PiperOrigin-RevId: 463434373
2022-08-08 08:11:22 +00:00
huangdarwin
a7a17dc2bb Rename findEncoderWithClosestFormatSupport.
Just a misc nit, since I found the name a bit confusing, and figured
findEncoderWithClosestSupportedFormat might be more descriptive.

PiperOrigin-RevId: 463433646
2022-08-08 08:10:23 +00:00
hschlueter
22822d8e19 Allow FrameProcessor.Factory to be set on Transformer.Builder.
Extract a FrameProcessor.Factory interface from GlEffectsFrameProcessor
and allow it to  be customized using a setter on Transformer.Builder.

PiperOrigin-RevId: 463433438
2022-08-08 08:09:33 +00:00
hschlueter
4adf199398 Replace Size with Pair in effects.
Size requires API 21. Using Pair instead will allow effects to be
used from API 18 during previewing once they are moved out of
transformer.

PiperOrigin-RevId: 463206474
2022-08-08 08:04:20 +00:00
hschlueter
5db7778ca3 Only recreate output EGLSurface when Surface changed.
PiperOrigin-RevId: 463193768
2022-08-08 08:03:28 +00:00
hschlueter
30e5bc9837 Allow stream offset passed to GlEffectsFrameProcessor to change.
This is needed for applying effects to a playlist.

The effects are applied based on the presentation time of the
frame in its corresponding media item and the offset is added
back before encoding.

Each time the offset changes, end of input stream is signalled
to the texture processors. This is needed because the texture
processors can expect monotonically increasing timestamp within
the same input stream but when the offset changes, the timstamps
jump back to 0.

PiperOrigin-RevId: 462714966
2022-08-08 07:59:56 +00:00
ibaker
2deb435625 Annotate methods that always return this with @CanIgnoreReturnValue
It's always safe to ignore the result of these methods, because the
caller already has a reference to the returned value.

PiperOrigin-RevId: 462388947
2022-08-08 07:55:25 +00:00
huangdarwin
2f977eeec9 HDR: Implement HLG EOTF and OETF.
This allows us to use BT.2020 RGB linear for intermediate shaders, which also
allows us to re-enable PeriodicVignetteProcessor, which should work properly in
linear color-spaces.

Manually tested by adding a GlEffectsWrapper, and confirming that HLG HDR editing still looks correct.

PiperOrigin-RevId: 462265821
2022-07-21 12:58:24 +00:00
olly
714edc93be Add ContrastProcessor for contrast adjustments.
PiperOrigin-RevId: 462232813
2022-07-21 12:57:07 +00:00
samrobinson
21016eaadf Don't include case statement if it falls through to default.
PiperOrigin-RevId: 461888238
2022-07-21 12:48:46 +00:00
huangdarwin
87198fe7f8 Only apply a MediaFormat-generated ColorInfo if it's valid.
Otherwise, invalid ColorInfo instances generated using faulty
MediaFormat#getInteger values could cause exceptions.

Confirmed that b/239435670 reproduces without this CL, and does not reproduce
with this CL.

PiperOrigin-RevId: 461862191
2022-07-21 12:46:55 +00:00
huangdarwin
ba9c9bb964 HDR: Use FP16 color representation for texture processors.
* Introduced `useHdr` for `GlEffect#toGlTextureProcessor`, so
  `TextureProcessor` implementations can decide how to handle HDR.
* Creating FP16 color textures for HDR input.

Tested via manual testing, adding a no-op GlEffectWrapper to the transformation to
force use of intermediate textures, adding a linear ramp to the fragment shader,
and trying to ascertain that there's a real reduction in posterization when
switching from 4-bit to 8-bit unsigned bytes, and again from 8-bit unsigned bytes
to 16-bit floating point.

PiperOrigin-RevId: 461613117
2022-07-21 12:42:32 +00:00
huangdarwin
9f7a159bc4 HDR: Throw when unexpected color transfer encountered.
This may happen when a containers' color transfer incorrectly does not match
the video's color transfer.

An example of a file with such a mismatch is the current Transformer demo HDR10
sample file.

Manually tested by confirming that no errors are emitted for SDR and HLG sample
files, and that errors are emitted for our incorrect HDR10 sample file.

PiperOrigin-RevId: 461583532
2022-07-21 12:41:33 +00:00
andrewlewis
604ab7fcda Make minor fixes to HDR handling
- Update profile selection logic to pick an HDR-compatible profile when doing HDR editing on H.264/AVC videos.
- Handle doing the capabilities check for all MIME types that support HDR (not just H.265/HEVC).
- Fix a bug where we would pass an HDR input color format to the encoder when using tone-mapping.
- Tweak how `EncoderWrapper` works so decisions at made at construction time.

Manually tested cases:
- Transformation of an SDR video.
- Transformation of an HDR video to AVC (which triggers fallback/tone-mapping on a device that doesn't support HDR editing for AVC).
- Transformation of an HDR video with HDR editing.

PiperOrigin-RevId: 461572973
2022-07-21 12:38:41 +00:00
samrobinson
b87fa45fea Add additional video files.
These are providing more variety and complexity.

All files are okay to be public.

PiperOrigin-RevId: 460935247
2022-07-15 10:19:56 +00:00
hschlueter
7954eeb3c2 Use COLOR_Format32bitABGR2101010 for HDR encoder configuration.
Also remove VideoEncoderSettings.colorProfile as there are no
concrete use cases for customizing this and it clashes with picking
the color format automatically based on SDR vs. HDR.

PiperOrigin-RevId: 460746987
2022-07-13 17:49:24 +00:00
samrobinson
f903869eb8 Fix assertion error when using high quality targeting API.
Add test that verifies SSIM with API enabled.

#minor-release

PiperOrigin-RevId: 460692420
2022-07-13 17:46:38 +00:00
samrobinson
a88426ae58 Clarify format is supported by encoder.
#cleanup
#minor-release

PiperOrigin-RevId: 460688226
2022-07-13 17:44:37 +00:00
claincly
e56219f1f6 Fix a mis-match in encoder priority.
PiperOrigin-RevId: 460500666
2022-07-13 17:39:16 +00:00
claincly
776c8a5544 Verified encoding performance, removing TODO.
PiperOrigin-RevId: 460459378
2022-07-12 15:15:03 +00:00
hschlueter
66e1229988 Set ColorInfo in decoder configuration format.
Pass the color info and HDR static metadata when configuring the decoder
using MediaFormatUtil.maybeSetColorInfo.

PiperOrigin-RevId: 460424985
2022-07-12 15:13:13 +00:00
hschlueter
a5ff4ef17f HDR: Check whether EXT_YUV_target extension is supported.
This extension is needed for editing HDR input with OpenGL, as the
ExternalTextureProcessor samples raw YUV values from the
external texture for HDR and converts them to RGB itself rather than
relying on the OpenGL driver to do this automatically as for SDR.

PiperOrigin-RevId: 460424154
2022-07-12 15:11:21 +00:00
hschlueter
7dc54efdb9 Merge MatrixTransformationProcessor and ExternalTextureProcessor.
This saves an intermediate texture copy step for use-cases
where matrix transformations are the first or only effects
in the chain.

PiperOrigin-RevId: 460239403
2022-07-12 15:09:22 +00:00
claincly
18f4068c06 Apply priority/operating rate settings for video encoding.
- Added setter to disable this feature.
- Added accompanying tests.
- Plan to run tests on the same set of settings on H265.

PiperOrigin-RevId: 460238673
2022-07-12 15:08:22 +00:00
samrobinson
91f1777741 Move Encoder quality API to VideoEncoderSettings.
Some other minor nits and adjustments to the API logic.

PiperOrigin-RevId: 459490431
2022-07-07 16:48:03 +00:00
huangdarwin
7078ce312d HDR: Remove ColorInfo.SDR constant
The SDR constant also specified a color space and range, in addition to
C.COLOR_TRANSFER_SDR. However, it turns out that SDR videos may use different color
space and range values, so following prior ExoPlayer conventions to have `null`
mean "generic SDR" is preferable here.

PiperOrigin-RevId: 459296746
2022-07-07 16:45:00 +00:00
huangdarwin
ab7747d953 HDR: Throw error if attempting HDR editing under API 31.
HDR editing is not supported under API 31

PiperOrigin-RevId: 459211106
2022-07-07 16:42:21 +00:00
huangdarwin
1869855c90 HDR: Remove unused EGL_GL_COLORSPACE_KHR attribute.
PiperOrigin-RevId: 459106221
2022-07-07 16:39:27 +00:00
samrobinson
7d2eb76019 Implement device based encoder bitrate mapping.
This feature is disabled by default for now.

PiperOrigin-RevId: 458932471
2022-07-04 19:57:36 +00:00
hschlueter
9c8dcb402b Fallback to SDR if encoder doesn't support HDR (HLG only).
If the input is HDR (HLG), check encoder capabilities for HDR support
and request tone-mapping to SDR during decoder configuration otherwise.
Capabilities are only checked for API 31 and above, as HDR editing is
not supported before.

As the encoder capabilities check needs to happen before selecting the
encoder to use (as this may depend on the resolution output by the
effects chain), the EncoderWrapper checks all candidate encoders
for the MIME type for HDR capabilities and only requests fallback to
SDR if none of them support it.

When the actual encoder is selected, the wrapper checks that it matches
one of the encoders is checked capabilities for.

PiperOrigin-RevId: 458511599
2022-07-04 19:55:45 +00:00
huangdarwin
a0870a42be HDR: Configure GL shaders and encoder.
Configure the GL shaders and encoder to take in HDR metadata.

This mostly just consists of passing the Format.colorInfo through
the VideoTranscodingSamplePipeline down to the encoder, rather than passing
the PQ-ness down to the GL step.

Due to b/237674316, this will remove HDR10+ support temporarily to introduce
support for HLG10.

Manually tested to confirm that HLG10 operations that don't affect color display
correctly after this CL with "HDR editing" in the demo checked, and continue to display incorrectly (as before this CL) without the option unchecked.

PiperOrigin-RevId: 458490810
2022-07-04 19:55:02 +00:00
claincly
deea5c927a Fix error in frame rate handling.
The old getString() will throw because FRAME_RATE can only be float or int.

PiperOrigin-RevId: 458481251
2022-07-04 19:54:06 +00:00
claincly
3df4f3eb19 Find only REGULAR_CODECS in EncoderUtil.
We used "ALL_COOECS" previously, and it is not necessary because "ALL_CODECS"
additionally the codecs that support tunneling/secure decoding, which there
is no use case in Transformer.

PiperOrigin-RevId: 458470278
2022-07-04 19:52:23 +00:00
claincly
1f47fa832c Round the frame rate in MediaFormat
Although MediaCodec claims supporting float frame rate, encoder init failed on
API21 Nexus 5. Since it's just a performance hint to the codec, it's OK to
generalize it to other API versions.

PiperOrigin-RevId: 458434650
2022-07-04 19:49:33 +00:00
hschlueter
cd0e5b99de Move DebugViewProvider out of Transformer class.
The GlEffectsFrameProcessor that will be part of the effects module
uses the DebugViewProvider. So it does not make sense for it
to be an inner interface of Transformer.

PiperOrigin-RevId: 458014932
2022-07-04 19:45:23 +00:00
hschlueter
234015cb95 FrameProcessor: Replace SurfaceInfo.Provider with setter.
The FinalMatrixTransformationProcessorWrapper ensures that the
surface is only replaced when it is not being rendered to and vice
versa.

PiperOrigin-RevId: 458007639
2022-07-04 19:44:37 +00:00
hschlueter
87beb273e4 Fix outputHeight pass-through condition.
The outputHeight in the TransformationRequest is the height of
the frame as it would be displayed (i.e., after applying any
rotation specified in the format). So pass-through should only
be used if the requested outputHeight matches the input
format's height after applying the rotation.

PiperOrigin-RevId: 457934867
2022-07-04 19:42:09 +00:00
hschlueter
b7b5f20e59 Allow changing input pixelWidthHeightRatio for GlEffectsFrameProcessor.
pixelWidthHeightRatio is now passed to setInputFrameInfo instead of
the factory.

PiperOrigin-RevId: 457696703
2022-07-04 19:39:42 +00:00
hschlueter
4819b28587 Rename setVideoFrameEffects to setVideoEffects.
PiperOrigin-RevId: 457023382
2022-06-27 10:44:17 +01:00
hschlueter
30c52c58f0 Allow FrameProcessor input surface size changes.
This will be useful for downgrading to a lower resolution during
a slow preview and for processing slide-shows once sequential
multi-asset editing is supported.

PiperOrigin-RevId: 457017255
2022-06-27 10:43:21 +01:00
hschlueter
f3893c146d Extract FrameProcessor interface from GlEffectsFrameProcessor.
PiperOrigin-RevId: 456814150
2022-06-27 10:42:00 +01:00
hschlueter
938d3c2e5b Disable flaky transformer emulator test on API 31.
videoEncoderFormatUnsupported_completesWithError() has recently
been flaky on API 31 emulators on presubmit because a different
exception than the expected exception is thrown.
This disables it on those emulators to reduce testing noise
until the underlying problem is investigated and resolved.

PiperOrigin-RevId: 456765512
2022-06-27 10:41:18 +01:00
hschlueter
709224fb1e Rename FrameProcessorChain to GlEffectsFrameProcessor.
This change is just renaming. There is no functional change intended.
The FrameProcessor interface will be created in a follow-up.

PiperOrigin-RevId: 456741628
2022-06-27 10:39:42 +01:00
samrobinson
352967f656 Add a Builder for DefaultEncoderFactory.
PiperOrigin-RevId: 456728032
2022-06-27 10:39:00 +01:00
hschlueter
69ab79418e Use GlTextureProcessor to avoid redundant copy in MediaPipeProcessor.
After this change GlEffects can use any GlTextureProcessor not just
SingleFrameGlTextureProcessor.
MediaPipeProcessor now implements GlTextureProcessor directly which
allows it to reuse MediaPipe's output texture for its output texture
and avoids an extra copy shader step.

PiperOrigin-RevId: 456530718
2022-06-27 10:37:35 +01:00
hschlueter
555ab97e34 Support chaining async GlTextureProcessors in FrameProcessorChain.
After this change, FrameProcessorChain chains any GlTextureProcessors
instead of only SingleFrameGlTextureProcessors.

The GlTextureProcessors are chained in a bidirectional manner using
ChainingGlTextureProcessorListener to feed input and output related
events forward and release events backwards.

PiperOrigin-RevId: 456478414
2022-06-27 10:36:51 +01:00
huangdarwin
f5d8800d51 FrameProcessor: Use factories instead of a builder for Presentation.
PiperOrigin-RevId: 456064021
2022-06-27 10:34:29 +01:00
hschlueter
981baae709 Implement chaining GlTextureProcessor.Listener.
In follow-ups the FrameProcessorChain will set an instance of this
listener for each GlTextureProcessor to chain it with its previous
and next GlTextureProcesssor.

PiperOrigin-RevId: 455628942
2022-06-27 10:32:38 +01:00
hschlueter
ee847d92c5 Wrap processor chain executor service for better exception handling.
The wrapper
* catches exceptions for each task and notifies the
  listener (this will be used more in follow-ups when processFrame
  is split into lots of listeners and callbacks),
* removes finished tasks from the queue and signals any exceptions
  that occurred to the listener each time a new task is executed.

PiperOrigin-RevId: 455345184
2022-06-16 11:23:44 +00:00
hschlueter
fdfca88019 Replace FrameProcessorChain#isEnded with listener method.
PiperOrigin-RevId: 455114693
2022-06-15 15:31:31 +00:00
hschlueter
ea7f1ca1e3 Use FrameProcessorChain#SurfaceProvider for encoder compat transform.
This change adds a SurfaceProvider interface which is necessary to
allow for texture processors whose output size becomes available
asynchronously in follow-ups.
VTSP's implementation of this interface wraps the encoder and provides
its input surface together with the output frame width, height, and
orientation as used for encoder configuration.
The FrameProcessorChain converts the output frames to the provided
orientation and resolution using a ScaleToFitTransformation and
Presentation replacing EncoderCompatibilityTransformation.

PiperOrigin-RevId: 455112598
2022-06-15 15:30:48 +00:00
hschlueter
a105d033a7 Check targetSdkVersion for frame dropping workaround.
Based on
https://developer.android.com/reference/android/media/MediaCodec#using-an-output-surface,
frame dropping behaviour depends on the target SDK version.
After this change transformer will only use
MediaFormat#KEY_ALLOW_FRAME_DROP if both the target and system SDK
version are at least 29 and default to its pre 29 behaviour where each
decoder output frame must be processed before a new one is rendered
to prevent frame dropping otherwise.

Also remove deprecated Transformer.Builder constructor without a
context and the context setter.

PiperOrigin-RevId: 453971097
2022-06-09 18:48:45 +00:00
hschlueter
cc1f32d094 Make GlUtil.GlException checked and remove flag to disable it.
Transformer always enabled glAssertionsEnabled, so there should
be no functional change.

ExoPlayer previously disabled glAssertionsEnabled, so GlUtil logged
GlExceptions instead of throwing them. The GlExceptions are now
caught and logged by the callers so that there should also be no
functional change overall.

This change also replaces EGLSurfaceTexture#GlException with
GlUtil#GlException.

PiperOrigin-RevId: 453963741
2022-06-09 18:47:46 +00:00
claincly
12d775f055 Ensure re-encode on performance tests
PiperOrigin-RevId: 453933854
2022-06-09 17:49:18 +00:00
huangdarwin
c5b881e089 Transformer GL: Split Presentation and Crop.
This removes the prior restriction of needing to remember not to crop and set aspect ratio in the same Presentation.Builder, and makes each class a bit more targeted.

This is partially made feasible by the past work to merge consecutive
MatrixTransformations into a single MatrixTransformationFrameProcessor, which
ensures that there's no loss in quality between successive MatrixTransformations.

PiperOrigin-RevId: 453660582
2022-06-09 17:44:04 +00:00
hschlueter
cdb8038782 Mention alpha for matrix transformation background color.
PiperOrigin-RevId: 453633920
2022-06-09 17:42:23 +00:00
hschlueter
457f446114 Implement default GlTextureProcessor in SingleFrameGlTextureProcessor.
SingleFrameGlTextureProcessor is now an abstract class containing a
default implementation of the more flexible GlTextureProcessor interface
while still exposing the same simple abstract methods for single frame
processing it previously did.

FrameProcessorChain and GlEffect will be changed to use
GlTextureProcessor in follow-ups.

PiperOrigin-RevId: 453633000
2022-06-09 17:41:09 +00:00
hschlueter
63436390de Add interface for async texture processors.
Implementations of this interface will be able to drop or add frames,
change timestamps, accept multiple input frames before producing
output, and process frames on their own background thread.

A default implementation of this interface will be added to SingleFrameGlTextureProcessor in a follow-up.

PiperOrigin-RevId: 453159835
2022-06-07 16:26:26 +00:00
hschlueter
35b5147eb1 Add async error listener to transformer to avoid exception wrapping.
This internal listener avoids wrapping the TransformationExceptions
in PlaybackExceptions that are handled via the Player.Listener and
is also used for FrameProcessingExceptions which already avoided
the PlaybackException layer previously.

This listener will also be useful in follow-ups for encoder-related
TransformationExceptions that are thrown in the SurfaceProvider that
will be called on the GL thread.

PiperOrigin-RevId: 452074575
2022-06-07 16:23:22 +00:00
hschlueter
87ab96d352 Move program initialization to texture processor constructor.
Once the more advanced GlTextureProcessor interface exists,
it will be possible to change the output size of a GlTextureProcessor
between frames. To keep the re-configuration based on the frame sizes
minimal, things indepedent of the frame size, such as the GlProgram,
can be initialized in the constructor.

PiperOrigin-RevId: 451997584
2022-05-31 10:43:47 +00:00
claincly
a629d09458 Skip BitrateAnalysisTest if device is incapable.
Add bitrate check to the "can encode" criteria.

PiperOrigin-RevId: 451868042
2022-05-30 16:51:51 +00:00
andrewlewis
b25d00a795 Retain stream time offsets through codecs
ExoPlayer applies a large time offset to buffers so that, if the input has negative timestamps, generally buffers seen by the decoders should have positive timestamps. Modify how the offset is handled in `Transformer` so that decoders and encoders generally see positive timestamps, by leaving the offset on samples when reading them in the base renderer (remove the code that removed the offset), and then removing the offset when muxing. Also update the frame processor chain and slow motion flattening code to retain the existing behavior after this change (these both need original media presentation timestamps)

Tested via existing end-to-end tests and manually verified that the overlay frame processor shows the expected original media timestamps.

Aside: we don't need the same logic as ExoPlayer to track stream offsets across the decoder yet, because we don't try to handle stream changes during playback in single asset editing. (There is an edge case of multi-period DASH that may not work but I doubt anyone will use that as input to `Transformer` before we change the code to handle multi-asset properly.) In future we should try to refactor interaction with the decoder to use the same code for Transformer and ExoPlayer.
PiperOrigin-RevId: 451846055
2022-05-30 16:45:15 +00:00
samrobinson
a1b89561c1 Treat pixels as unsigned and correct pixel count division.
PiperOrigin-RevId: 451428202
2022-05-30 16:42:38 +00:00
huangdarwin
67354cf2a7 Transformer: Add 8k24fps transform test.
Most devices won't support 8k decoding, so they'll skip this test entirely.

As the video is quite short, this test shouldn't be any longer than the nearby,
long-running 4k60 test.

PiperOrigin-RevId: 451423368
2022-05-30 16:41:41 +00:00
samrobinson
6c4f6ecf46 Clarify variables and improve documentation of SSIM.
PiperOrigin-RevId: 451392021
2022-05-30 16:39:50 +00:00
samrobinson
224761833f Use updated test name in BUILD and docs.
PiperOrigin-RevId: 451384408
2022-05-30 16:39:01 +00:00
hschlueter
9649411a4f Use video passthrough if clipping starts at key frame.
PiperOrigin-RevId: 451380267
2022-05-30 16:37:19 +00:00
hschlueter
3e8a2f104c Expand SSIM skipping to all Nexus API 21 devices.
PiperOrigin-RevId: 451371681
2022-05-30 16:36:28 +00:00
huangdarwin
099a542dac Transformer: Skip SSIM in tests when fallback is applied.
SSIM calculation requires the input and output dimensions to be identical.

For devices that can't encode the input dimensions, skip SSIM calculations and
log the cause. Only apply this on tests where the encoder may not support the
input file dimensions.

PiperOrigin-RevId: 451364904
2022-05-30 16:35:35 +00:00
hschlueter
fe3831c5b4 Fix handling clipping in transformer renderers.
Decode-only video frames (needed when the frame at / first frame after the
clipping start is not a key frame) need to be decoded but not passed to
the frame processor chain or encoder.

The clipping start offset needs to be removed from the frame timestamps
in the passthrough and video pipelines.
There are no changes needed for this in the audio pipeline, as it doesn't
use the input timestamps -- it uses its own timestamps derived from the
buffer sizes instead.

Also add demo option to try this out.

#minor-release

PiperOrigin-RevId: 451353609
2022-05-30 16:34:38 +00:00
hschlueter
0eaf3d30c8 Rename GlFrameProcessor to SingleFrameGlTextureProcessor.
Also update names of implementations to match design doc.
In follow-ups, SingleFrameGlTextureProcessor will become
an abstract implementation of a new GlTextureProcessor
interface.

Texture processor makes sense as it processes OpenGL textures.
The term frame processor will be used for something else in
follow-ups.

PiperOrigin-RevId: 451142085
2022-05-30 16:31:13 +00:00
hschlueter
de1dceedc1 Skip SSIM calculation on Nexus 5 API 21.
There is a problem with the ImageReader formats used by the
SSIM helper that only occurs for Nexus 5 API 21, so as a workaround
we can skip the SSIM calculation on Nexus 5 API 21.

This skips just the SSIM calculation (by setting the value to
1.0 instead and logging). The tests still run when SSIM is skipped
so that we can detect other failures.

PiperOrigin-RevId: 450903183
2022-05-26 11:56:20 +00:00
hschlueter
678b6c0438 Replace 640x360 H264 test video with 320x240 H264 video.
Decoding and encoding 320x240 H264 video should be supported
on all devices from Android 5.0 based on CDD requirements.
https://source.android.com/compatibility/5.0/android-5.0-cdd#5_2_video_encoding

640x360 encoding doesn't seem to be supported on Nexus 5.

PiperOrigin-RevId: 450901715
2022-05-26 11:55:21 +00:00
hschlueter
d840aa4dc1 Skip transform4K60 test if 4K decoding is not supported.
4K decoding is not supported (not required to be supported) on all
devices, e.g., Nexus 5 does not support it.

PiperOrigin-RevId: 450682519
2022-05-26 11:50:24 +00:00
hschlueter
196a99aa5f Don't reallocate EGLSurface for same debug surface.
Recreating an EGLSurface for a surface that already has an
EGLSurface is not allowed according to the
[documentation](https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglCreatePlatformWindowSurface.xhtml).

This fix was tested on the devices listed in the bug
description (Pixel 5a, Nexus 5).

PiperOrigin-RevId: 450473569
2022-05-23 18:37:04 +01:00
claincly
d6a6529b06 Skip performance tests if device is not capable.
PiperOrigin-RevId: 450427828
2022-05-23 14:54:02 +01:00
andrewlewis
ce1ac5894d Track validity of debug SurfaceView
The debug surface view's output surface can become invalid during a transformation due to the parent activity pausing, for example. This can currently cause a crash when backing out of the `TransformerActivity` in the demo because the surface can be destroyed before the transformer has fully canceled.

Also clarify naming of the outputSurface and inline the private method that created `EGLSurface`s (it was shorter after removing the debug preview).

PiperOrigin-RevId: 449963440
2022-05-24 11:03:09 +01:00
samrobinson
2a45dd3930 Rename expected to reference in SsimHelper.
Reference is clearer when used in conjunction with actual, and matches
other SSIM documentation.

PiperOrigin-RevId: 449486177
2022-05-24 10:57:56 +01:00
andrewlewis
dcb58c767c Tidy releasing FrameProcessorChain
PiperOrigin-RevId: 449238525
2022-05-24 10:52:33 +01:00
andrewlewis
0e2bb8c8bd Update stale bug reference.
The old reference was just for prototyping HDR. The new reference is for planned work to use the correct formats for input and output for HDR editing in GL.

PiperOrigin-RevId: 449211792
2022-05-24 10:48:57 +01:00
hschlueter
d59186e53c Combine multiple matrix transformations in one shader.
When using a MatrixTransformationFrameProcessor per transformation
matrix, each frame processor's shader applies the matrix to the
vertices and clips the result to the NDC range when drawing the
output frame.
This change combines consecutive MatrixTransformations into a single
MatrixTransformationFrameProcessor by multiplying the individual
matrices while updating and clipping the visible polygon after
each matrix and mapping the resulting visible polygon back to the
input space so that its vertices and the combined transformation
matrix can be used in the shader.

PiperOrigin-RevId: 448521068
2022-05-24 10:44:15 +01:00
samrobinson
de33eee6ec Increase test timeout for 4k60 regression test.
Pixel 5 was occasionally reaching the 2 minute limit.

PiperOrigin-RevId: 448244803
2022-05-24 10:40:25 +01:00
huangdarwin
5067a8da0d Transformer GL: Update test name reference.
The prior test does not exist, and is not a pixel test.

PiperOrigin-RevId: 448224929
2022-05-24 10:37:51 +01:00
samrobinson
d72ffe4ba1 Expose decoder name to analysis files.
PiperOrigin-RevId: 447950623
2022-05-24 10:29:06 +01:00
samrobinson
8d2597acc6 Force encoding for the on-device regression tests.
transformWithDecodeEncode is now transformWithoutDecodeEncode,
to test the passthrough path.

PiperOrigin-RevId: 447796892
2022-05-24 10:26:19 +01:00
samrobinson
f8f1a4d1ed Expose the codec name to the analysis files.
PiperOrigin-RevId: 447724152
2022-05-10 17:52:05 +01:00
hschlueter
63dcdf5803 Add listener for FrameProcessingExceptions.
This listener replaces
FrameProcessorChain#getAndRethrowBackgroundExceptions.
The listener uses a new exception type FrameProcessingException
separate from TransformationException as the frame processing
components will be made reusable outside of transformer soon.

PiperOrigin-RevId: 447455746
2022-05-10 17:47:22 +01:00
hschlueter
2e544224c2 Fix GlUtil vector size constant name.
This constant is used for https://docs.gl/es2/glVertexAttribPointer
which takes the number of components per generic vertex attribute
(meaning the size of the individual coordinate vectors here) not the
number of attributes (the number of vertices that the old constant
name referred to).

PiperOrigin-RevId: 447427241
2022-05-10 17:42:39 +01:00
samrobinson
ff7629aba5 Match the suggested bitrate to the actual Kush Gauge formula.
As defined in Kush Amerasinghe's paper 'H.264 For the rest of us'.

PiperOrigin-RevId: 446988272
2022-05-09 11:05:43 +01:00
huangdarwin
e60ba54ddf Transformer GL: Remove obsolete TODOs.
We won't try to provide/rethrow helpful error messages that are already
provided by GL, as this sort of task would expand into writing a GL verifier.
A Gl verifier is unnecessarily complex for minimal payoff, especially as Apps
expected to read GL error messages would mostly be those writing custom
GlFrameProcessors, who should be already be familiar with reading GL error
messages anyways.

PiperOrigin-RevId: 446950837
2022-05-09 11:04:05 +01:00
hschlueter
e9919f6da1 Remove redundant attribute from matrix transformation vertex shader.
The texture coordinates can be derived from the frame coordinates.

PiperOrigin-RevId: 446770538
2022-05-09 11:01:33 +01:00
hschlueter
4b8ff72d30 Use MatrixTransformation instead of wrapping its GlFrameProcssor.
ScaleToFitFrameProcessor, PresentationFrameProcessor,
and EncoderCompatibilityFrameProcessor now each implement
MatrixTransformation instead of wrapping
MatrixTransformationFrameProcessor.

PiperOrigin-RevId: 446480286
2022-05-09 10:58:56 +01:00
hschlueter
c94035278c Separate matrix effect specification and implementation.
This change splits AdvancedFrameProcessor into 4 files:
- MatrixTransformationFrameProcessor for the GlFrameProcessor
  implementation
- MatrixTransformation and GlMatrixTransformation for the GlEffect
  specification
- MatrixUtils for the static matrix helpers

PiperOrigin-RevId: 446236384
2022-05-09 10:51:14 +01:00
hschlueter
b410a922fe Introduce GlEffect interface for effect specification.
PiperOrigin-RevId: 446143537
2022-05-09 10:44:30 +01:00
hschlueter
3869c0a851 Return recommended output size for FrameProcessorChain.
Also add warning for what can happen if this is ignored.

Previously the return value was overridden by setOutputSurface, now
setOutputSurface only overrides the values using internally.

PiperOrigin-RevId: 445377036
2022-05-09 10:38:32 +01:00
huangdarwin
0de9955eb2 Transformer: Misc nits
* Add a /* paramName= */ comment
* Remove an unnecessary comma.
* Remove extra "internal " for a b/### link

PiperOrigin-RevId: 445169649
2022-05-09 10:30:18 +01:00
claincly
c82cbc094f Change video asset used in TranscodeQualityTest
The old video has a `irregular' resolution of 1080x720. Change to a 1920x1080
video.

PiperOrigin-RevId: 445124065
2022-05-09 10:28:43 +01:00
huangdarwin
0f3e63909c Transformer: Restrict frame buffer size for all omx decoders API >= 29.
PiperOrigin-RevId: 445119411
2022-05-09 10:27:58 +01:00
claincly
564dc11f28 Add device capability based test skipping.
Change test name prefix from `transform` to `transcode` for consistency.

PiperOrigin-RevId: 445106224
2022-05-09 10:25:07 +01:00
huangdarwin
f269963e89 Transformer GL: Add support for pixelWidthHeightRatio.
To ensure frame processor operations operate on square pixels,
make the frame taller or wider for non-square input pixels.

In addition to automated tests, this was tested by changing the
inputFormat.pixelWidthHeightRatio in the TransformerVideoRenderer.

PiperOrigin-RevId: 444553517
2022-05-09 10:11:18 +01:00
huangdarwin
22c9b8168d Transformer: remove unused bitmap test option.
Remove unused saveTestBitmapToCacheDirectory throwOnFailure option.

#cleanup

PiperOrigin-RevId: 444516857
2022-04-26 12:05:30 +01:00
huangdarwin
3cf7f58edf Transformer GL: save actual bitmap before checking pixel diffs.
This allows the actual bitmap to be saved, even if output
dimensions are different than expected. Otherwise, differing
output dimensions would throw an exception, preventing the bitmap
from being saved.

PiperOrigin-RevId: 444512210
2022-04-26 11:37:03 +01:00
samrobinson
c87d03e78e Add 30 second high motion clips to AndroidTestUtil.
30 second clips at higher resolutions are useful for testing encoder
behaviour compared to shorter clips.

PiperOrigin-RevId: 444505865
2022-04-26 10:59:07 +01:00
huangdarwin
f1cda5fe2a FrameProcessor: Avoid early rounding in Presentation output dimensions.
PiperOrigin-RevId: 444253425
2022-04-25 14:38:17 +01:00
hschlueter
02743456a0 Use last frame processor to draw on debug surface view.
The last frame processor could use a different drawing command
than a four vertex triangle strip. So we need call its drawing
method again instead of assuming the shape.

PiperOrigin-RevId: 444217274
2022-04-26 15:04:33 +01:00
huangdarwin
36d190bb66 Transformer GL: Avoid no-op ScaleToFit / Presentation FrameProcessors
Avoid no-op FrameProcessors in VideoTranscodingSamplePipeline, to avoid
creating resources for and executing GL for a no-op vertex+fragment shader.

EncoderCompatibilityFrameProcessor will still always be exercised.

PiperOrigin-RevId: 443675833
2022-04-26 15:03:22 +01:00
andrewlewis
c8744ee9bd Add an SSIM test with 360p input
This test should run on all devices from API 21 (the media uses Baseline
profile level 3.0 H.264) to give us coverage of the full pipeline (forcing
re-encoding) and SSIM calculation on all devices.

PiperOrigin-RevId: 443650002
2022-04-26 15:00:34 +01:00
andrewlewis
0a378a4555 Fix encoder configuration before API 25
Float I-frame intervals are only supported from API 25 onwards (see https://developer.android.com/reference/android/media/MediaFormat#KEY_I_FRAME_INTERVAL).

Setting a null value for profile/level can cause a native crash when configuring the decoder.

PiperOrigin-RevId: 443647924
2022-04-26 14:59:44 +01:00
claincly
f9bce2a6b6 Remove a completed TODO item
PiperOrigin-RevId: 443620931
2022-04-26 14:58:37 +01:00
huangdarwin
3cfdfb41c6 FrameProcessor: Create EncoderCompatibilityFrameProcessor.
Split rotationDegrees changes to EncoderCompatibilityFrameProcessor, a new
FrameProcessor.

This removes automatic rotationDegrees adjustments from Presentation, which
allows Presentation to be used for changes before the end of a
FrameProcessorChain pipeline.

PiperOrigin-RevId: 443387226
2022-04-26 14:57:40 +01:00
andrewlewis
27c1876c9f Expand Samsung/1+ workaround to API 31 + 32
The issue with decoding getting stuck is still reproducible on `samsung/beyond1lteeea/beyond1:12/SP1A.210812.016/G973FXXUEGULB:user/release-keys`.

PiperOrigin-RevId: 443368546
2022-04-26 14:55:39 +01:00
hschlueter
6939464f91 Don't use API 26 SurfaceTexture constructor in frame processor tests.
After this change the test will use EGL_NO_SURFACE or a pixel buffer surface if using no surface is not supported.

PiperOrigin-RevId: 443113794
2022-04-26 14:53:31 +01:00
claincly
f903e4a9f8 Apply resolution fix.
Some devices under report their resolution support, like 2144 for 2160 in
H265, 1072 for 1080 in H264. This CL only takes care of these two cases,

- reporting 1920x1080 is supported when the device reports 1920x1072, and
- reporting 3840x2160 is supported when the device reports 3840x2144

PiperOrigin-RevId: 443095042
2022-04-26 14:50:35 +01:00
samrobinson
d1f8e96d9c Add 4 different video resolutions for bitrate analysis.
PiperOrigin-RevId: 442844189
2022-04-26 14:48:20 +01:00
huangdarwin
7f7e84bee3 FrameProcessor: Improve PresentationFrameProcessor javadocs.
PiperOrigin-RevId: 442835160
2022-04-26 14:47:21 +01:00
hschlueter
5248bbb882 Add context param to GlFrameProcessor initialize().
PiperOrigin-RevId: 442826391
2022-04-26 14:46:24 +01:00
hschlueter
afa551ab8f Block until FrameProcessorChain is released.
This is safer because it will prevent any future problems with
creating a new FrameProcessorChain before the previous one has
completed its async release.

From [eglDestroyContext documentation](https://www.khronos.org/registry/EGL/sdk/docs/man/html/eglDestroyContext.xhtml):
"If the EGL rendering context is not current to any thread,
eglDestroyContext destroys it immediately."
The context isn't current to any thread here because GlUtil calls
eglMakeCurrent with EGL_NO_CONTEXT before calling eglDestroyContext.
So everthing should be released once the FrameProcessorChain's
release task terminates.

PiperOrigin-RevId: 442807484
2022-04-26 14:45:31 +01:00
claincly
7a67609561 Add checks for device capability in tests.
PiperOrigin-RevId: 442751310
2022-04-26 14:42:31 +01:00
huangdarwin
3f4d03e731 FrameProcessor: Add checks to ensure width and height are positive.
Negative and zero values should be disallowed.

PiperOrigin-RevId: 441757246
2022-04-26 14:39:29 +01:00
huangdarwin
29e32540de FrameProcessor: Move test bitmaps to a separate directory.
* Group what's now many related test PNGs by moving them to their own directory.
* Move bitmap references to files where they're used, as each bitmap is only
used once each, except the original bitmap.

PiperOrigin-RevId: 441485489
2022-04-26 14:36:31 +01:00
hschlueter
22e016e851 Clarify TransformationRequest setResolution javadoc.
Scaling and rotation using ScaleToFitFrameProcessor may change the
the resolution and aspect ratio, so defaulting to the same as the
input is no longer accurate.

PiperOrigin-RevId: 441463349
2022-04-26 14:35:12 +01:00
hschlueter
f25c912c72 Rename GlFrameProcessor updateProgramAndDraw to drawFrame.
What a minimal implementation should include is now explained in the
interface javadoc while the method name reflects what the method does.

PiperOrigin-RevId: 441432059
2022-04-26 14:34:11 +01:00
huangdarwin
34f014ce5c FrameProcessor: Add aspect ratio changes to Presentation.
PiperOrigin-RevId: 441250773
2022-04-26 14:31:14 +01:00
olly
83daa052cb Rename TracksInfo and TrackGroupInfo
1. TracksInfo is renamed to Tracks
2. TracksInfo.TrackGroupInfo is renamed to Tracks.Group

PiperOrigin-RevId: 441232373
2022-04-26 14:30:14 +01:00
hschlueter
af8435816a Make TransformerEndToEndTest constants private.
PiperOrigin-RevId: 441230941
2022-04-26 14:29:19 +01:00
samrobinson
36c5b9407c Remove transformer from inputValues json.
Transformer resolves to null when wrapped, this is a debug line
that was not removed.

PiperOrigin-RevId: 441200790
2022-04-26 14:28:13 +01:00
hschlueter
9b4aba33a3 Fix FrameProcessorChain method ordering.
signalEndOfInputStream makes more sense before isEnded and
getPendingInputFrameCount is related to registerInputFrame.

PiperOrigin-RevId: 441134418
2022-04-26 14:26:17 +01:00
samrobinson
0a16813f1c Allow suppression of AssertionError in TransformerAndroidTestRunner.
PiperOrigin-RevId: 440952335
2022-04-26 14:25:15 +01:00
samrobinson
8521d48f86 Fix incorrect test id.
PiperOrigin-RevId: 440906834
2022-04-26 14:24:21 +01:00
claincly
803cf64a06 Removing using the test asset that has Mp3 audio track.
The stream with the mp3 track is added because

- We only encode to AAC
- We only encode when the source track is not AAC

Now that we have a way to force encoding, we no longer need the mp3 track.

The test asset is kept for later parameterized testing.

PiperOrigin-RevId: 440876080
2022-04-26 14:23:26 +01:00
hschlueter
20547c3392 Add WorkerThread annotation to FrameProcessorChain helpers.
PiperOrigin-RevId: 440347550
2022-04-26 14:20:31 +01:00
huangdarwin
8a7fcb4eae FrameProcessor: Add setCrop to Presentation.
PiperOrigin-RevId: 440325693
2022-04-26 14:19:38 +01:00
claincly
1ca80741e5 Change SSIM transcode MIME type to H264.
The MIME type was set to H265 to force transcoding. Now that we have an encoder
factory that forces encoding, switching back to H264 ensures the quality test
is conducted on more devices (those don't support H265 can be tested now).

However, H265 should be part of the quality test after we have proper mechanism
to skip test based on device capability.

PiperOrigin-RevId: 440132471
2022-04-26 14:18:35 +01:00
claincly
7d0034748f Add trying 1/3 resolution in fallback.
1/4 is probably unnecessary, as 1/3 of 4k is 720, which should be supported on
all practical encoders.

PiperOrigin-RevId: 440055144
2022-04-26 14:14:33 +01:00
claincly
a2d92ec623 Change SSIM test threshold from .95 to .90.
We don't currently have enough understanding of the correlation between a
specific SSIM score and video quality. Dropping to .90 to make most tests pass.
Especially when there's no discernible difference from the videos with .9 and
.95 SSIM.

PiperOrigin-RevId: 440047551
2022-04-26 14:13:31 +01:00
claincly
1da7fd8956 Add test to query device capabilities.
PiperOrigin-RevId: 439861685
2022-04-26 14:11:27 +01:00
claincly
8bdce6ade6 Preallocate buffer and use byte for Luminance in SSIM.
PiperOrigin-RevId: 439855702
2022-04-26 14:09:52 +01:00
claincly
7c64b1915c Add test skipping for API < 23 for EncoderPerformanceTest.
PiperOrigin-RevId: 439802597
2022-04-26 14:07:48 +01:00
hschlueter
c235e4f447 Add matrix provider for AdvancedFrameProcessor and examples in demo.
The matrix provider allows the transformation matrix to be updated
for each frame based on the timestamp.

The following example effects using this were added to the demo:
* a zoom-in transition for the start of the video,
* cropping a rotating rectangular frame portion,
* rotating the frame around the y-axis in 3D.

PiperOrigin-RevId: 439791592
2022-04-26 14:06:36 +01:00
hschlueter
f75710be93 Fix typo in AdvancedFrameProcessorTest.
PiperOrigin-RevId: 439599201
2022-04-06 12:00:19 +01:00
claincly
5b258ef8ab Add javadoc for SSIM helper.
PiperOrigin-RevId: 439332549
2022-04-06 11:56:02 +01:00
hschlueter
8138a9f48f Support android.opengl.Matrix in AdvancedFrameProcessor.
This allows apps to use AdvancedFrameProcessor to apply transformations
in 3D space. This functionality is not used in transformer otherwise.

PiperOrigin-RevId: 439313406
2022-04-06 11:54:59 +01:00
claincly
5b77625582 Add test to evaluate performance related encoding parameters.
PiperOrigin-RevId: 439268235
2022-04-06 11:53:53 +01:00
hschlueter
de00030f98 Merge GlFrameProcessor#setInputSize() and initialize().
PiperOrigin-RevId: 439266087
2022-04-06 11:52:05 +01:00
hschlueter
d3931d8b96 Add ExternalCopyFrameProcessor to frameProcessors list.
PiperOrigin-RevId: 438847583
2022-04-06 11:49:31 +01:00
hschlueter
2a66c7b8f5 Move FrameProcessorChain OpenGL setup to factory method.
The encoder surface is no longer needed for the OpenGL setup and frame
processor initialization, as a placeholder surface is used instead. So
all of the setup can now be done in the factory method.

PiperOrigin-RevId: 438844450
2022-04-06 11:47:45 +01:00
samrobinson
af5386cbc1 Add the frame count to TransformationResult.
Calculate throughputFps for TransformationTestResult.

PiperOrigin-RevId: 438817440
2022-04-06 11:44:16 +01:00
samrobinson
0c6882867b Add file logging for skipping instrumentation tests.
PiperOrigin-RevId: 438808231
2022-04-06 11:43:16 +01:00
hschlueter
fa47233628 Add FrameProcessorChain factory method and make constructor private.
PiperOrigin-RevId: 438804850
2022-04-06 11:41:04 +01:00
hschlueter
6d753afc4c Store max timestamp rather than last written timestamp per track.
This allows the MuxerWrapper to keep using trackTypeToTimeUs for
calculating the video duration but slightly changes the meaning of
its interleaving constraints.

PiperOrigin-RevId: 438780686
2022-04-06 11:40:06 +01:00
claincly
12fc9d1070 Move a commonly used encoder factory mode to util.
The encoder factory will be used in other tests.

PiperOrigin-RevId: 438552381
2022-04-06 11:35:43 +01:00
hschlueter
70cffd8cca Split configureOutputSize into setInputSize and getOutputSize.
This makes it easier (smaller CL diff) to merge output size
configuration and initialize() in a follow-up.

PiperOrigin-RevId: 438543247
2022-04-06 11:34:34 +01:00
samrobinson
d8e5e2de7a Re-use test runner in each loop of RepeatedTranscodeTransformationTest
PiperOrigin-RevId: 438542239
2022-04-06 11:33:38 +01:00
hschlueter
839f342e55 Use placeholder surface to configure OpenGL and frame processors.
The placeholder surface is either EGL_NO_SURFACE or a 1x1 pbuffer
depending on whether the device supports EGL_KHR_surfaceless_context.

PiperOrigin-RevId: 438541846
2022-04-06 11:32:38 +01:00
hschlueter
aadbf3d59b glClear in FrameProcessorChain so the GlFrameProcessors don't have to.
Since the output textures and surfaces are managed by the
FrameProcessorChain, clearing them there makes sense.
This is also less error-prone as it might not be obvious to
someone implementing a GlFrameProcessor that they need to
glClear. (Clearing twice won't cause any problems.)

PiperOrigin-RevId: 438532247
2022-04-06 11:30:23 +01:00
samrobinson
01c24e4de8 Move Json helper methods to AndroidTestUtil.
PiperOrigin-RevId: 438253138
2022-04-06 11:26:29 +01:00
samrobinson
e699765df5 Add support for analyzing bitrates across devices.
Allows for input values to be propagated to the analysis file.

PiperOrigin-RevId: 438030322
2022-04-06 11:24:26 +01:00
hschlueter
d97de5b9f0 Use microseconds not nanoseconds for GlFrameProcessor.
This requires an additional nanos to micros conversion because
the SurfaceTexture uses nanos. But as the timestamps from the
MediaCodec decoder (propagated in DefaultCodec#releaseOutputBuffer) are
in microseconds no precision is lost here.

Also add test that checks output video duration.

PiperOrigin-RevId: 438010490
2022-04-06 11:22:22 +01:00
hschlueter
bd6eaab6b4 Reword MAXIMUM_AVERAGE_PIXEL_ABSOLUTE_DIFFERENCE javadoc.
MAXIMUM_AVERAGE_PIXEL_ABSOLUTE_DIFFERENCE was copied from a test
class, but BitmapTestUtil isn't a test. So the javadoc needs
rewording to reflect that.

PiperOrigin-RevId: 438001833
2022-04-06 11:20:17 +01:00
hschlueter
808909289c Use @linkplain for link text that doesn't match symbol name.
PiperOrigin-RevId: 437992927
2022-04-06 11:18:03 +01:00
andrewlewis
d2a9419ad3 Add support for requesting color transfer to SDR
From Android T onwards `MediaCodec` supports requesting tone-mapping down to
SDR. Add an option to request this behavior and document that it isn't
supported before T. Also add an option in the demo app to try it out.

Tested manually on a prerelease build.

PiperOrigin-RevId: 437765325
2022-04-06 11:10:06 +01:00
andrewlewis
79db98e733 Remove unnecessary initialization
PiperOrigin-RevId: 437753013
2022-04-06 11:09:03 +01:00
hschlueter
827cf51dc9 Check thread name for GL methods.
The thread name is used to verify the thread in both  createOpenGlObjectsAndInitializeFrameProcessors() and processFrame().
Also remove glThread field that was only used for this verification.

PiperOrigin-RevId: 437730804
2022-04-06 11:07:45 +01:00
hschlueter
92eb09fa6f Fix output viewport size of empty FrameProcessorChain.
Since the output size can be overridden, the viewport should be
ouputWidth/Height and NOT the ExternalCopyFrameProcessor's output size
which matches the input size.

PiperOrigin-RevId: 437256635
2022-04-06 11:01:43 +01:00
hschlueter
5f573ad3ad Remove old TODOs and reassign another TODO.
* The AdvancedFrameProcessor calls use() in updateProgramAndDraw().
* The AdvancedFrameProcessor has the same input and output dimensions.

PiperOrigin-RevId: 437231350
2022-03-25 14:10:47 +00:00
hschlueter
0f5686fe07 Allow frame processors to be set on Transformer.Builder.
Also make GlFrameProcessor, ScaleToFitFrameProcessor, and
AdvancedFrameProcessor public.

PiperOrigin-RevId: 437227388
2022-03-25 14:08:41 +00:00
hschlueter
d5056072a8 Check for GL errors after glClear and glDrawArrays.
These methods can cause errors so we should check for them.
https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glClear.xml
https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glDrawArrays.xml

PiperOrigin-RevId: 437218519
2022-03-25 14:07:49 +00:00
andrewlewis
117456c137 Improve throughput on devices requiring workaround
On some devices, decoding gets stuck when the number of frames pending at the
`SurfaceTexture` is too high. We added a workaround that only allows one frame
to be pending at a time. That fixed the issue, however, based on on-device
testing it seems that it's safe to queue more than one frame.

Add a method that returns a safe estimate of the number of frames that can be
pending at a time, and use this to limit the number of frames that can be
released from the decoder but not processed by the frame processor chain.

PiperOrigin-RevId: 437057075
2022-03-25 14:05:08 +00:00
hschlueter
37559deacf Configure the frame sizes in FrameProcessorChain instead of caller.
Configuring the frame sizes between frame processors is now the
FrameProcessorChain's rather than the caller's responsibility.
The caller can getOutputSize() and override it for encoder fallback
in configure().

PiperOrigin-RevId: 437048436
2022-03-25 14:04:19 +00:00
hschlueter
20daaa20ef Move OpenGL setup to FrameProcessorChain#configure().
The factory method is replaced by a public constructor and
configure() method which configures the input/output surfaces
and handles the OpenGL setup.

This is a prerequisite for removing the responsibility of the
caller to configureSizes() before creating the chain in a follow-up.

PiperOrigin-RevId: 437028882
2022-03-25 14:03:17 +00:00
huangdarwin
2a363ac3fc Transformer: Always use FrameProcessorChain when decoding.
This allows us to bypass many device-specific issues, that only occur when
decoding directly to an encoder surface, without OpenGL. This also allows us
to maintain fewer code branches, which require additional testing to verify
correctness.

PiperOrigin-RevId: 437003138
2022-03-24 15:42:12 +00:00
huangdarwin
5bc94da16a FrameProcessor: Create PresentationFrameProcessor.
Allow apps to modify how frames are presented, via modifying resolution.

A follow-up CL will provide aspect ratio, cropping, etc.

PiperOrigin-RevId: 436963312
2022-03-24 11:42:05 +00:00
hschlueter
c5e8503e2c Increase test frame processing wait time.
PiperOrigin-RevId: 436961627
2022-03-24 11:29:52 +00:00
hschlueter
831866f441 Add javadoc for more FrameProcessorChain fields.
PiperOrigin-RevId: 436723149
2022-03-23 13:36:22 +00:00
samrobinson
76d44bec30 Remove SSIM calculation on SEF videos.
SEF is similar to frame editing, where the input and output videos are
not intended to match, so SSIM will not provide a reliable value. To
check SSIM correctly in future, we would need to provide
golden/expected video files to compare output against.

PiperOrigin-RevId: 436707240
2022-03-23 11:56:30 +00:00
hschlueter
bda6604757 Remove FrameEditor reference.
PiperOrigin-RevId: 436702840
2022-03-23 11:25:58 +00:00
andrewlewis
89098a8a83 Remove analysis error suppression
PiperOrigin-RevId: 436470453
2022-03-22 14:42:02 +00:00
hschlueter
e14b8b0c6e Remove unnecessary 'final' from local variables.
PiperOrigin-RevId: 436469107
2022-03-22 14:35:19 +00:00
hschlueter
ecfbc65a0d Convert FrameEditor to a FrameProcessorChain.
The FrameProcessorChain manages a List<GlFrameProcessor>.
FrameProcessorChainDataProcessingTest now tests chaining ScaleToFit-
and AdvancedFrameProcessors.

PiperOrigin-RevId: 436468037
2022-03-22 14:29:53 +00:00
huangdarwin
57953d8947 FrameProcessor: Make a class member private final.
PiperOrigin-RevId: 436211808
2022-03-21 15:21:17 +00:00
samrobinson
a60e5f597b Force video encoding if VideoEncoderSettings are not default.
Add a MH test exercising this behaviour.

PiperOrigin-RevId: 436177198
2022-03-21 12:16:50 +00:00
huangdarwin
2a14e3c604 FrameProcessor: Add a ScaleToFitFrameProcessor builder.
This allows us to input scale and rotation in an easier-to-use manner.

PiperOrigin-RevId: 436175982
2022-03-21 12:08:18 +00:00
andrewlewis
0292f82ddf Fix end of stream detection for SSIM helper
PiperOrigin-RevId: 436165237
2022-03-21 11:00:21 +00:00
andrewlewis
9e62ea3fca Move image buffer extraction to test thread
This also ensures that if there's an error reading the image data then this
gets surfaced as an analysis exception.

PiperOrigin-RevId: 435680785
2022-03-18 17:55:26 +00:00
claincly
12543a9682 Split method findEncoderWithClosestFormatSupport.
Add checking for bitrate mode settings.
Add logging to the encoder filtering.

PiperOrigin-RevId: 435662418
2022-03-18 16:43:26 +00:00
andrewlewis
db0093f4c8 Remove AMR NB encoding from SSIM quality test
This currently causes the test to fail on Pixel 6 Pro running a recent S build
SQ1D.220205.004.

There is no need to test audio transcoding while we are measuring video
quality.

PiperOrigin-RevId: 435635314
2022-03-18 14:42:49 +00:00
hschlueter
96f8771de0 Always use FrameEditor for Nexus 5 and Moto Z Play.
PiperOrigin-RevId: 435628703
2022-03-18 14:10:07 +00:00