Commit graph

693 commits

Author SHA1 Message Date
tofunmi
521c210fd1 Create Timestamp iterator
PiperOrigin-RevId: 558738035
2023-08-22 15:27:42 +01:00
samrobinson
99ac6feeee Add varargs convenience constructors to Sequence & Composition.Builder
Improves API ease-of-use & readability and reduces verbose boilerplate.

PiperOrigin-RevId: 558163557
2023-08-22 15:17:55 +01:00
samrobinson
2db6f0aee7 Ensure audio components check incoming data is valid.
Default PCM encoding is only set for decoders outputting raw.

Tests migrated to abide by tighter restrictions.

PiperOrigin-RevId: 558129452
2023-08-18 15:33:44 +01:00
sheenachhabra
15650c6bf3 Add buildUpon method in Composition
We will need to build upon original composition
for pause and resume feature and will have to set modified sequences.
In the next CL, will add method to set sequences on builder and will
probably remove it from the constructor parameter (to avoid providing
two APIs for same thing).

PiperOrigin-RevId: 558127334
2023-08-18 15:32:28 +01:00
huangdarwin
030d8148ba Test: Add debug info to debug test failure
I wasn't able to reproduce the error in the bug by running videoTranscoding_withTextureInput_completesWithCorrectFrameCountAndDuration
100 times, but I figured this added debug info may help the next time the
issue is seen.

PiperOrigin-RevId: 558118443
2023-08-18 15:31:00 +01:00
huangdarwin
86678b99ce Decoder: Add SM-F721 to HLG blocklist.
Not only restricted to SM-F721B, per other [Models](https://www.gsmarena.com/samsung_galaxy_z_flip4-11538.php#:~:text=SM%2DF721B%2C%20SM%2DF721B/DS%2C%20SM%2DF721N%2C%20SM%2DF721U%2C%20SM%2DF721U1%2C%20SM%2DF721W).

PiperOrigin-RevId: 557857282
2023-08-18 15:17:48 +01:00
tofunmi
dd0f88490c Add ability to queue bitmap by timestamp to the sampleConsumer
PiperOrigin-RevId: 557837922
2023-08-18 15:16:24 +01:00
samrobinson
c740b58efa Add API for injecting AudioMixer.Factory.
Rename AudioMixerImpl to DefaultAudioMixer.

Removes the AudioMixerImpl specific getOutputAudioFormat, as the caller
defines and sets this.

PiperOrigin-RevId: 555887722
2023-08-11 20:26:21 +00:00
claincly
7e12d7619a Avoid create short lived object
PiperOrigin-RevId: 555204528
2023-08-10 12:27:00 +00:00
claincly
e8a18e208a Improve DebugTraceUtil logging
PiperOrigin-RevId: 554872827
2023-08-10 12:10:50 +00:00
christosts
cba027c3b4 Util.getMaxPendingFramesCountForMediaCodecDecoders(): Remove unused arguments
PiperOrigin-RevId: 554838494
2023-08-10 12:05:08 +00:00
samrobinson
cc141f7bf4 Use specific type for SequenceAssetLoader internal logic.
PiperOrigin-RevId: 554431024
2023-08-07 11:36:43 +00:00
samrobinson
82387ccfe6 Remove unneeded check against silence generator remaining.
PiperOrigin-RevId: 553771328
2023-08-07 11:33:37 +00:00
claincly
af546b8eda Fix decoder is queued buffer with timestamp TIME_END_OF_SOURCE
ExoPlayer queues the EOS buffer to the decoder with offset/size/timestamp all
equal to zero, and a EOS flag.

69769c77b3 set TIME_END_OF_SOURCE on the EOS buffer from the extractor.
Queueing the EOS buffer to the decoder with TIME_END_OF_SOURCE causes some
decoders to output wrong timestamps in its output.

This CL replicates what ExoPlayer does in Transformer.

PiperOrigin-RevId: 553104213
2023-08-07 11:24:11 +00:00
samrobinson
b15e7553e5 Add the hasSource method to the AudioMixer interface.
PiperOrigin-RevId: 551804976
2023-08-01 14:19:04 +01:00
claincly
c221958889 Remove setInputFrameInfo
After this change, every queued bitmap is treated as an individual input stream
(like a new MediaItems).

This change merges the FrameDropTest and FrameDropPixelTest into one (while maintaining all the test cases)

- This is accomplished by generating bitmaps with timestamps on it in FrameDropTest and compare them with goldens (one may call this a pixel test, please lmk if you want this to be renamed)
- The most part of the change comes from DefaultVideoFrameProcessorVideoFrameRenderingTest. The overall working is
  - We bypass the input manager
  - The TestFrameGenerator generates frames based on timestamps. In this case, we generate frames with timestamps on it
  - The generated frame is sent to texture output and in turn saved to bitmaps
  - We then compare the generated bitmap with the goldens

PiperOrigin-RevId: 551795770
2023-08-01 14:17:59 +01:00
samrobinson
ccdde8806e Account for endTimeUs when outputting with no sources.
When endTimeUs is set, restrict output to that point.

PiperOrigin-RevId: 551555392
2023-08-01 14:15:34 +01:00
claincly
e77c318d96 Create Util method for SparseArray.contains()
And fixed usages in Transformer and Effect.

PiperOrigin-RevId: 551484860
2023-08-01 14:12:10 +01:00
samrobinson
b7d7027424 Update Composition javadoc for expanded audio support.
PiperOrigin-RevId: 550922961
2023-08-01 14:08:46 +01:00
andrewlewis
9a2bfc4042 Prioritize H.265 and H.264 over other formats
This should make no functional difference because `SampleExporter` always
checks for H.265 and H.264 first. However, in case we ever change that code,
these are used in priority order so it's better to order them accordingly.

PiperOrigin-RevId: 550894935
2023-08-01 14:05:42 +01:00
samrobinson
54093a152e Integrate AudioMixer for audio export.
Adds support for compositions with multiple audio sequences.

PiperOrigin-RevId: 550880626
2023-08-01 14:03:22 +01:00
samrobinson
357c458028 Ensure EOS is queued after processing generated silence with effects.
When generating silence for AudioProcessingPipeline, audio never
queued EOS downstream.

Linked to this, when silence followed an item with audio, the silence
was added to SilentAudioGenerator before the mediaItem reconfiguration
occurred. If the silence had effects, the APP would be flushed after
silence queued EOS, resetting APP.isEnded back to false, so AudioGraph
never ended.

Regression tests reproduce failure without fix, but pass with it.

PiperOrigin-RevId: 550853714
2023-08-01 14:02:09 +01:00
huangdarwin
35bee3299c Remove VisibleForTesting visibility in 8k workaround.
PiperOrigin-RevId: 550843914
2023-08-01 14:01:02 +01:00
Googler
465e7c1277 Rewrite tracing to be more flexible and provide more info.
For each event, the timestamp and presentation time is logged. The trace can
then be dumped to a tsv file and easily imported in a spreadsheet.

PiperOrigin-RevId: 550839156
2023-08-01 13:59:51 +01:00
sheenachhabra
c2615a679f Skip tests if device needs disable 8k workaround
PiperOrigin-RevId: 549967326
2023-08-01 13:47:26 +01:00
samrobinson
beceb996ac Create ExportException for audio from UnhandledAudioFormatException.
All audio processing export errors are related to a UAFE - passing
audioFormat separately is redundant.

PiperOrigin-RevId: 549427685
2023-07-20 10:35:45 +01:00
claincly
177187e237 Allow multiple inputs to the processing graphs
The new logic covers the old single-sequence use cases.

PiperOrigin-RevId: 549307918
2023-07-20 10:33:35 +01:00
samrobinson
035934c6d4 Combine audio mixing logic across Transformer.
AudioMixingUtil#mix handles input & output in float or Int16 PCM. Given
Float and Int16 use different sample ratnes, this util handles
conversion between the two, based on the encoding being mixed to.

Migrate AudioMixer to use the util, removing AudioMixingAlgorithm
interface and implementation. ChannelMixingAudioProcessor will be
migrated after additional performance checks.

PiperOrigin-RevId: 548994584
2023-07-20 10:22:58 +01:00
claincly
18033c9c1b Add basic SingleInputVideoGraph by wrapping DefaultVideoFrameProcessor
SingleInputVideoGraph implements GraphInput now, so the asset loaders would
interface directly with SIVG, rather than VideoSampleExporter. This is to pave
way for multi-asset video processing.

PiperOrigin-RevId: 547561042
2023-07-13 15:57:47 +01:00
claincly
e81ae0a8eb Remove stale (incorrect) javadoc lines from transformer.java
PiperOrigin-RevId: 547451166
2023-07-13 15:51:28 +01:00
claincly
bbdc64a732 Remove stale TODO
The said test is added in `TransformerMixedInputEndToEndTest` and
`TransformerSequenceEffectTest`.

PiperOrigin-RevId: 547407374
2023-07-13 15:48:58 +01:00
samrobinson
4983d4f339 Switch argument order passed to scaleLargeTimestamp.
durationUs is almost always going to be a larger number than the sample
rate, so pass as the main value, rather than the multiplier.

PiperOrigin-RevId: 547193927
2023-07-13 15:44:15 +01:00
andrewlewis
fcadecf2b1 Add note recommending OpenGL tone-mapping
PiperOrigin-RevId: 547192957
2023-07-13 15:43:01 +01:00
andrewlewis
7df09fe4b7 Disable codec tone-mapping on Fold 4 API 33
The codec plugin sets crop values incorrectly.

PiperOrigin-RevId: 547160088
2023-07-13 15:39:21 +01:00
claincly
ddd000128f Add an Input interface to SampleExporter.
PiperOrigin-RevId: 546822962
2023-07-13 15:27:40 +01:00
claincly
9c74e78f33 Revert DefaultCodec to final
The class is made non-final for a test in 12584fbb20.

PiperOrigin-RevId: 546718462
2023-07-13 15:23:07 +01:00
andrewlewis
a0bc8d95c2 Improve texture asset loader documentation
Add documentation for threading requirements at the class level (in
addition to existing documentation on the methods) to improve
discoverablility. Also fix a couple of nits in the javadoc (US English
spelling, avoid passive voice) and in `OnInputFrameProcessedListener`.

PiperOrigin-RevId: 546303732
2023-07-13 15:14:21 +01:00
claincly
50c6efe95d Rename SamplePipeline to SampleExporter
PiperOrigin-RevId: 545974776
2023-07-13 15:06:10 +01:00
huangdarwin
99a143a74e Transformer: Add 8k disabling workaround to avoid timeouts.
PiperOrigin-RevId: 545970477
2023-07-13 15:04:59 +01:00
claincly
372cd7f952 Remove unused constructor parameter
PiperOrigin-RevId: 545909062
2023-07-13 15:00:12 +01:00
huangdarwin
756845426b HDR: Add SM-X900 to HLG10 tone mapping blocklist.
PiperOrigin-RevId: 545647083
2023-07-05 13:59:58 +00:00
samrobinson
c33a17d89c Wrap AudioSamplePipeline input and processing within AudioGraph.
This is a no-op refactor.

PiperOrigin-RevId: 545421300
2023-07-05 09:20:12 +00:00
kimvde
8ea79a13f9 Add a TODO to make TransformationRequest.Builder package private
PiperOrigin-RevId: 545400150
2023-07-05 09:19:06 +00:00
kimvde
81c83e4213 Remove HDR modes from TransformationRequest
PiperOrigin-RevId: 545352306
2023-07-05 09:16:52 +00:00
samrobinson
f60f79bb10 Handle media item (Effects/Format) changes in AudioSamplePipeline.
On a MediaItem change, the input Format (and Effects to apply) may be
different. Therefore the AudioProcessingPipeline must be reconfigured
to determine what processing is active, and what the AudioFormat of the
data output is. In the event that it is different, additional
AudioProcessor instances must be used to ensure the encoder will still
be able to accept the audio buffers.

PiperOrigin-RevId: 544338451
2023-06-29 23:14:10 +00:00
kimvde
32bd5dd718 Fix forceInterpretHdrAsSdr not being set in DefaultAssetLoaderFactory
PiperOrigin-RevId: 544296301
2023-06-29 23:13:00 +00:00
andrewlewis
81fc9ea509 Make DefaultDecoderFactory visible
This fixes a broken javadoc link and makes it possible to customize media
source creation via the asset loader factory without also having to
implementing a decoder factory from scratch.

PiperOrigin-RevId: 543995538
2023-06-29 23:06:47 +00:00
kimvde
a632118875 Remove TransformationRequest.Builder deprecated methods
TransformationRequest.Builder will become package private.

PiperOrigin-RevId: 543987255
2023-06-29 23:03:41 +00:00
claincly
f98a10f3f2 Move GlShaderProgram creation away from VFP creation
PiperOrigin-RevId: 543773418
2023-06-29 23:00:11 +00:00
kimvde
f8491fc61f Deprecate Transformer.Builder.setTransformationRequest()
Usages will be removed in follow-up changes.

PiperOrigin-RevId: 543654397
2023-06-29 22:55:29 +00:00