mirrors/gst-plugins-rs
mirrors/gst-plugins-rs
1
1
Fork 0
mirror of https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs.git synced 2024-06-08 17:29:22 +00:00
Code Issues Projects Releases Wiki Activity
3284 commits 13 branches 86 tags 24 MiB
Commit graph

90 commits

Author SHA1 Message Date
Liam b4fd6cf362 aws: Add system-defined metadata options to both sinks 
Add to awss3sink and awss3putobjectsink elements the following
paramerters which are set on the uploaded S3 objects:

* cache-control;
* content-encoding; and
* content-language

Bugfix: Set the content-type and content-disposition values in the S3
putobject call. Previously the params were defined on the element but
unused.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1585>
 2024-05-27 10:25:22 +03:00
Sebastian Dröge 539000574b aws: Update to base32 0.5 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1576>
 2024-05-17 07:50:51 +00:00
Sebastian Dröge be3ae583bc Fix new Rust 1.78 clippy warnings 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1559>
 2024-05-02 18:36:23 +03:00
Sanchayan Maity a3e30b499f aws: Introduce a property to use path-style addressing 
AWS SDK switched to virtual addressing as default instead of path
style earlier. While MinIO supports virtual host style requests,
path style requests are the default.

Introduce a property to allow the use of path style addressing if
required.

For more information, see
https://github.com/minio/minio/blob/master/docs/config/README.md#domain
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1527>
 2024-04-10 00:23:22 +00:00
François Laignel a870d60621 aws: improve error message logs 
The `Display` and `Debug` trait for the AWS error messages are not very useful.

- `Display` only prints the high level error, e.g.: "service error".
- `Debug` prints all the fields in the error stack, resulting in hard to read
  messages with redudant or unnecessary information. E.g.:

> ServiceError(ServiceError { source: BadRequestException(BadRequestException {
> message: Some("1 validation error detected: Value 'test' at 'languageCode'
> failed to satisfy constraint: Member must satisfy enum value set: [ar-AE,
> zh-HK, en-US, ar-SA, zh-CN, fi-FI, pl-PL, no-NO, nl-NL, pt-PT, es-ES, th-TH,
> de-DE, it-IT, fr-FR, ko-KR, hi-IN, en-AU, pt-BR, sv-SE, ja-JP, ca-ES, es-US,
> fr-CA, en-GB]"), meta: ErrorMetadata { code: Some("BadRequestException"),
> message: Some("1 validation error detected: Value 'test' at 'languageCode'
> failed to satisfy constraint: Member must satisfy enum value set: [ar-AE,
> zh-HK, en-US, ar-SA, zh-CN, fi-FI, pl-PL, no-NO, nl-NL, pt-PT, es-ES, th-TH,
> de-DE, it-IT, fr-FR, ko-KR, hi-IN, en-AU, pt-BR, sv-SE, ja-JP, ca-ES, es-US,
> fr-CA, en-GB]"), extras: Some({"aws_request_id": "1b8bbafd-5b71-4ba5-8676-28432381e6a9"}) } }),
> raw: Response { status: StatusCode(400), headers: Headers { headers:
> {"x-amzn-requestid": HeaderValue { _private: H0("1b8bbafd-5b71-4ba5-8676-28432381e6a9") },
> "x-amzn-errortype": HeaderValue { _private:
> H0("BadRequestException:http://internal.amazon.com/coral/com.amazonaws.transcribe.streaming/") },
> "date": HeaderValue { _private: H0("Tue, 26 Mar 2024 17:41:31 GMT") },
> "content-type": HeaderValue { _private: H0("application/x-amz-json-1.1") },
> "content-length": HeaderValue { _private: H0("315") }} }, body: SdkBody {
> inner: Once(Some(b"{\"Message\":\"1 validation error detected: Value 'test'
> at 'languageCode' failed to satisfy constraint: Member must satisfy enum value
> set: [ar-AE, zh-HK, en-US, ar-SA, zh-CN, fi-FI, pl-PL, no-NO, nl-NL, pt-PT,
> es-ES, th-TH, de-DE, it-IT, fr-FR, ko-KR, hi-IN, en-AU, pt-BR, sv-SE, ja-JP,
> ca-ES, es-US, fr-CA, en-GB]\"}")), retryable: true }, extensions: Extensions {
> extensions_02x: Extensions, extensions_1x: Extensions } } })

This commit adopts the most informative and concise solution I could come up
with to log AWS errors. With the above error case, this results in:

> service error: Error { code: "BadRequestException", message: "1 validation
> error detected: Value 'test' at 'languageCode' failed to satisfy constraint:
> Member must satisfy enum value set: [ar-AE, zh-HK, en-US, ar-SA, zh-CN, fi-FI,
> pl-PL, no-NO, nl-NL, pt-PT, es-ES, th-TH, de-DE, it-IT, fr-FR, ko-KR, hi-IN,
> en-AU, pt-BR, sv-SE, ja-JP, ca-ES, es-US, fr-CA, en-GB]",
> aws_request_id: "a40a32a8-7b0b-4228-a348-f8502087a9f0" }

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1521>
 2024-03-26 20:05:32 +01:00
François Laignel 9f27bde36a aws: use fixed BehaviorVersion 
Quoting [`BehaviorVersion` documentation]:

> Over time, new best-practice behaviors are introduced. However, these
> behaviors might not be backwards compatible. For example, a change which
> introduces new default timeouts or a new retry-mode for all operations might
> be the ideal behavior but could break existing applications.

This commit uses `BehaviorVersion::v2023_11_09()`, which is the latest
major version at the moment. When a new major version is released, the method
will be deprecated, which will warn us of the new version and let us decide
when to upgrade, after any changes if required. This is safer that using
`latest()` which would silently use a different major version, possibly
breaking existing code.

[`BehaviorVersion` documentation]: https://docs.rs/aws-config/1.1.8/aws_config/struct.BehaviorVersion.html

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1520>
 2024-03-26 17:44:16 +01:00
Philippe Normand be12c0a5f7 Fix clippy warnings after upgrade to Rust 1.77 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1512>
 2024-03-21 17:33:32 +00:00
Sebastian Dröge 4ad101b53b Use once_cell crate directly again 
The glib crate does not depend on it anymore and also does not re-export
it anymore.

Also switch some usages of OnceCell to OnceLock from std.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1441>
 2024-01-31 18:07:57 +02:00
Arun Raghavan 06213714c5 aws: putobjectsink: Fix a couple of minor log typos 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1416>
 2024-01-11 15:38:36 -05:00
Arun Raghavan 6d47045a60 aws: s3sink: Fix spelling of debug category 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1337>
 2023-12-18 16:13:48 -05:00
Arun Raghavan 410d104ad6 aws: s3putobjectsink: Add a flush-on-error property 
Makes sure we can send out data even if the pipeline shutdown in error.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1337>
 2023-12-18 16:13:48 -05:00
Arun Raghavan 12dbf50ddc aws: s3putobjectsink: Add some thresholds for flushing 
Lets us connect when we perform a flush

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1337>
 2023-12-18 16:13:48 -05:00
Arun Raghavan a54b2dd39e aws: s3: Add a new awss3putobjectsink 
When streaming small amounts of data, using awss3sink might not be a
good idea, as we need to accumulate at least 5 MB of data for a
multipart upload (or we flush on EOS).

The alternative, while inefficient, is to do a complete PutObject of
_all_ the data periodically so as to not lose data in case of a pipeline
failure. This element makes a start on this idea by doing a PutObject
for every buffer.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1337>
 2023-12-18 10:39:23 -05:00
Sebastian Dröge 1d9c89e3fe Update to AWS SDK 0.101 / 0.59 / 0.38 2023-11-20 10:13:13 +02:00
Sebastian Dröge dee27e35b7 Update to latest AWS SDK 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1395>
 2023-11-17 11:22:29 +02:00
Mathieu Duponchelle 5371eb52ad Port to AWS SDK 0.57/0.35 
Co-authored-by: Sebastian Dröge <sebastian@centricular.com>
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1379>
 2023-11-03 15:13:45 +00:00
Sebastian Dröge 16b917abb1 Update for gst::Rank API changes 2023-11-02 14:10:59 +02:00
Sebastian Dröge 16c00ae3f5 Set sync=false in rsfilesink / s3sink 
BaseSink defaults to sync=true and that doesn't make much sense for
these elements.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1376>
 2023-10-30 17:38:46 +02:00
Sebastian Dröge 855b03a9ea Use let-else instead of match for weak reference upgrades 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1375>
 2023-10-30 11:34:35 +02:00
Arun Raghavan bb26e04a55 aws: s3: Properly percent-decode GstS3Url 
We previously only percent-decoded the first fragment. This doesn't
necessarily harm anything, but for consistency we keep the structure
un-encoded, and encode when converting to a string representation.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1333>
 2023-10-24 12:52:12 -04:00
Arun Raghavan 51129febeb aws: s3sink: Fix handling of special characters in key 
Properly URL-encode the string if needed, and add some tests for a
couple of cases.

Fixes: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/431
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1333>
 2023-10-24 12:52:12 -04:00
Jordan Petridis a2d7f42138 Fix compilation after glib bindings changes 
loggable_error! can now expand variables and we no longer need
the format! on our side.

https://github.com/gtk-rs/gtk-rs-core/pull/1210

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1366>
 2023-10-22 01:20:56 +03:00
Sebastian Dröge 8d433761d1 Fix indentation of let-else blocks 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1308>
 2023-08-29 12:21:38 +00:00
Bilal Elmoussaoui dd2d7d9215 Use re-exported once_cell 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1268>
 2023-07-06 17:50:49 +03:00
Bilal Elmoussaoui 2cc98bf410 Adapt to glib::Continue rename 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1268>
 2023-07-06 17:50:49 +03:00
Seungha Yang 1f0ce101eb awstranscriber: Tone down log message 
It's not an ERROR case at all

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1263>
 2023-06-28 23:57:54 +09:00
Mathieu Duponchelle 6346d5608e net/aws/transcriber: track discont offset in input stream 
and add it up to subsequent transcripts.

This ensures synchronization is maintained even after the input stream
experiences a discontinuity and a gap in its timestamps.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1230>
 2023-06-02 08:55:11 +00:00
François Laignel 7ba0073052 use Pad builders for optional name definition 
Also, apply auto-naming in the following cases

* When building from a non wildcard-named template, the name of the template is
  automatically assigned to the Pad. User can override with a specific name by
  calling `name()` on the `PadBuilder`.
* When building with a target and no name was provided via the above, the
  GhostPad is named after the target.

See https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/issues/448
Auto-naming discussion: https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/-/merge_requests/1255#note_1891181

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1197>
 2023-05-12 12:55:31 +02:00
Mathieu Duponchelle f366c20869 awstranscriber: fix what we send over for translations 
Prior to this commit, we were sending over words concatenated together
with no separators, for instance "Idon'twanttobeanemperor".

The translation service seems clever enough to translate the contents
anyway, but there is no reason to make its task harder than necessary,
and it didn't re-add separators when the target language was the same as
the source language, which resulted in less than ideal output.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1171>
 2023-04-10 20:47:12 +00:00
Mathieu Duponchelle 408fd2030c awstranscriber: slight debug improvement 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1171>
 2023-04-10 20:47:12 +00:00
Guillaume Desmottes 403004a85e fix typos 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1170>
 2023-04-10 13:35:32 +02:00
Seungha Yang 762fb86ce7 awstranscriber: Reset start_time per task 
Otherwise wrong start time can be assigned if the element is
reused with state change

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1159>
 2023-04-05 18:22:59 +00:00
Seungha Yang 4000d60305 awstranscriber: Avoid too large initial GAP event 
Initialized GstSegment.position is always zero

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1154>
 2023-04-03 13:05:15 +00:00
Sebastian Dröge 6fe806c2b5 aws: Update to AWS SDK 0.55/0.25 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1152>
 2023-03-31 09:12:26 +00:00
François Laignel 2b32d00589 net/aws/transcriber: use two queues for sending transcript items 
* A queue dedicated to transcript items not intended for translation.
* A queue dedicated to transcript items intended for translation. The items are
  enqueued after a separator is detected or translate-lookahead was reached.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 20:29:31 +01:00
François Laignel 5a5ca76d9d net/aws/transcriber: desambiguify SrcPad output items queue 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 12:41:07 +01:00
François Laignel 162db2f3b9 net/aws/transcriber: fix translate lookahead 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 12:39:15 +01:00
François Laignel d5d6a4daf9 net/aws/transcriber: rename prop transcript-lookahead & TranslationSrcPad 
... as translate-lookahead and TranslateSrcPad.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1137>
 2023-03-16 12:37:31 +01:00
François Laignel 3b3f0c1a29 net/aws/transcriber: fix transcript-lookahead prop nick 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1136>
 2023-03-14 21:11:33 +01:00
François Laignel 299e25ab3c net/aws/transcriber: translate: optional experimental translation tokenization 
This commit adds an optional experimental translation tokenization feature.
It can be activated using the `translation_src_%u` pads property
`tokenization-method`. For the moment, the feature is deactivated by default.

The Translate ws accepts '<span></span>' tags in the input and adds matching
tags in the output. When an 'id' is also provided as an attribute of the
'span', the matching output tag also uses this 'id'.

In the context of close captions, the 'id's are of little use. However, we can
take advantage of the spans in the output to identify translation chunks, which
more or less reflect the rythm of the input transcript.

This commit adds simples spans (no 'id') to the input Transcript Items and
parses the resulting spans in the translated output, assigning the timestamps
and durations sequentially from the input Transcript Items. Edge cases such as
absence of spans, nested spans were observed and are handled here. Similarly,
mismatches between the number of input and output items are taken care of by
some sort of reconcialiation.

Note that this is still experimental and requires further testings.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1109>
 2023-03-14 13:48:32 +00:00
François Laignel 743e97738f net/aws/transcriber: add translation request src pads 
This commit adds an optional transcript translation feature implemented as
request src Pads.

When requesting a src Pad, the user can specify the translation language code
using Pad properties 'language-code'.

The following properties are defined on the Element:

- 'transcribe-latency': formerly 'latency', defines the expected latency for
  the Transcribe webservice.
- 'translate-latency': defines the expected latency for the Translate
  webservice.
- 'transcript-lookahead': maximum transcript duration to send to translation
  when a transcript is hitting its deadline and no punctuation was found.

When the input and output languages are the same, only the 'transcribe-latency'
is used for the Pad. Otherwise, the resulting latency is the addition of
'transcribe-latency' and 'translate-latency'.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1109>
 2023-03-14 13:48:32 +00:00
François Laignel b9cd71d8eb net/aws/transcriber: fix eos not being sent 
For eos to be sent from the srcpad task loop, we need to go through `dequeue`.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1122>
 2023-03-09 13:07:03 +01:00
François Laignel 2ea9f147ab net/aws/transcriber: fix deadlock when the pipeline is interrupted 
... also makes sure to abort the taks_iter Future.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1122>
 2023-03-09 13:07:03 +01:00
Sebastian Dröge 3ef8a48ded Fix a few new clippy warnings 
Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1120>
 2023-03-07 08:47:01 +00:00
François Laignel 4a988aaeb8 net/aws/transcriber: use a TranscriberLoop struct 
This helps gather together the details related to the `TranscriberLoop`.
One difference with previous implementation is that the ws `Client` is
build each time the loop is started instead of being reused. With the new
approach, we don't keep the connection open after EOS and we should be
more resistant in case of a connection failure.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel f1a080c94e net/aws/transcriber: own transcription items 
So that we can avoid copying the content.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel 36ae29d746 net/aws: enqueue transcribed buffers within the ws loop 
Instead of sending transcription events to the src pad loop, this commit
enqueues the transcribed buffers immediately in the ws loop, then notifies
the src pad loop. The src pad loop is only in charge of dequeuing the buffers.

This should help with upcoming evolutions.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel 00153754bb net/aws: use aws-sdk-transcribestreaming 
Switch from manual webservice client impl to `aws-sdk-transcribestreaming`.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
François Laignel 57f365979c net/aws: remove aws_ from the aws_transcribe* folder names 
Those folders reside under `aws`, so there's shouldn't be any confusion.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1104>
 2023-03-01 08:47:58 +00:00
rajneeshksoni 994c79569e awss3sink: Add properties to set content-Type and content-disposition. 
for uploaded object default content-type is set to binary/octet-stream,
which is correct.
metadata cannot be used to set content-type and content-disposition as
setting metadata add a prefix x-amz-meta to key
e.g. setting metadate "content-type=video/mp4" actually set value as
x-amz-meta-content-type. So these has to be seaprate property.

Part-of: <https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1085>
 2023-02-09 19:04:07 +00:00