Commit Graph

402 Commits (30ea88591b728cca0896018dbf67c2298070c669)

Author SHA1 Message Date
bashonly 149eb0bbf3
[extractor/youtube] Fix `uploader_id` extraction
Closes #6247
Authored by: bashonly
2 years ago
Bruno Guerreiro 78a78fa74d
[extractor/youtube] Add hyperpipe instances (#6020)
Authored by: Generator
2 years ago
Roland Hieber 05799a48c7
[extractor/youtube] Update invidious and piped instances (#6030)
Authored by: rohieb
2 years ago
Simon Sawicki 6839ae1f6d
[utils] `traverse_obj`: Fix more bugs
and cleanup uses of `default=[]`

Continued from b1bde57bef
2 years ago
pukkandan b032ff0f03
[extractor/youtube] Handle `consent.youtube` 2 years ago
pukkandan dad2210c0c
[extractor/youtube] Support `/live/` URL 2 years ago
mzhou 253ac4ba6a
[extractor/youtube] Retry manifest refresh for live-from-start (#5670)
Avoids ending download early when live stream is temporarily offline.
Best used with somewhat large `--retry-sleep extractor:` and `--extractor-retries`

Authored by: mzhou
2 years ago
pukkandan 08e29b9f1f
[cleanup] Misc
Closes #5576, closes #5887
2 years ago
pukkandan 9bb856998b
[extractor/youtube] Extract DRC formats 2 years ago
Matthew c733555106
[extractor/youtube:tab] Extract metadata from channel items (#5569)
Authored by: coletdjnz
2 years ago
pukkandan 71eb82d1b2
[extractor/youtube] Subtitles cannot be translated to `und`
Closes #5674
2 years ago
Bnyro bc87dac75f
[extractor/youtube] Add `piped.video` (#5571)
Closes #5518
Authored by: Bnyro
2 years ago
pukkandan 9f14daf22b
[extractor] Deprecate `_sort_formats` 2 years ago
pukkandan 6368e2e639
[cleanup] Misc
Closes #5541
2 years ago
pukkandan a4894d3e25
[extractor/youtube] Consider language in format de-duplication 2 years ago
pukkandan 171a31dbe8
[extractor] Add a way to distinguish IEs that returns only videos 2 years ago
pukkandan a8c754cc00
[extractor/youtube] Fix bug in handling of music URLs
Bug in bd7e919a75
Closes #5502
2 years ago
pukkandan 08270da5c3
[extractor/youtube] Fix `ytuser:` 2 years ago
pukkandan bd7e919a75
[extractor/youtube:tab] Improvements to tab handling (#5487)
* Better handling of direct channel URLs - See https://github.com/yt-dlp/yt-dlp/pull/5439#issuecomment-1309322019
* Prioritize tab id from URL slug - Closes #5486
* Add metadata for the wrapping playlist
* Simplify redirect for music playlists
2 years ago
Matthew e72e48c53f
[extractor/youtube] Ignore incomplete data error for comment replies (#5490)
When --ignore-errors is used.
Closes https://github.com/yt-dlp/yt-dlp/issues/4669
Authored by: coletdjnz
2 years ago
Matthew 0cf643b234
[extractor/youtube] Differentiate between no and disabled comments (#5491)
`comments` and `comment_count` will be set to None, as opposed to 
an empty list and 0, respectively.

Fixes https://github.com/yt-dlp/yt-dlp/issues/5068

Authored by: coletdjnz, pukkandan
2 years ago
Matthew 4dc23a8051
[extractor/youtube:tab] Fix video metadata from tabs (#5489)
Closes #5488
Authored by: coletdjnz
2 years ago
Matthew 86973308cd
[extractor/youtube:tab] Update tab handling for redesign (#5439)
Closes #5432, #5430, #5419
Authored by: coletdjnz, pukkandan
2 years ago
Bruno Guerreiro e14ea7fbd9
[extractor/youtube] Update piped instances (#5441)
Closes #5286
Authored by: Generator
2 years ago
Matthew 6141346d18
[extractor/youtube] Update playlist metadata extraction for new layout (#5376)
Fixes https://github.com/yt-dlp/yt-dlp/issues/5373

Authored by: coletdjnz
2 years ago
pukkandan 2e30b46fe4
[extractor/youtube] Improve chapter parsing from description
Closes #5448
2 years ago
nosoop 9da6612b0f
[extractor/youtube] Fix `duration` for premieres (#5382)
Closes #5378
Authored by: nosoop
2 years ago
coletdjnz e63faa101c
[extractor/youtube] Fix `live_status` extraction for playlist videos
Regression in 867c66ff97

Authored by: coletdjnz
2 years ago
bsun0000 5318156f1c
[extractor/youtube] Mark videos as fully watched
Closes #2555
Authored by: bsun0000
2 years ago
pukkandan d5d1df8afd
[cleanup Misc
Closes #5162
2 years ago
pukkandan 6678a4f0b3
[extractor/youtube] Fix live_status
Bug in 4d37720a0c
2 years ago
pukkandan 5225df50cf
[extractor/youtube:tab] Let `approximate_date` return timestamp 2 years ago
pukkandan 0468a3b325
[jsinterp] Improve separating regex
Fixes https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1273974909
2 years ago
Matthew 867c66ff97
[extractor/youtube] Extract concurrent view count for livestreams (#5152)
Adds new field `concurrent_view_count`
Closes https://github.com/yt-dlp/yt-dlp/issues/4843

Authored by: coletdjnz
2 years ago
Lesmiscore 4d37720a0c
[extractor/youtube] Download `post_live` videos from start (#5091)
* The fragments are generated as a `LazyList`. So only the required formats are expanded during download, but all fragment lists are printed/written in infojson.
* The m3u8 formats which cannot be downloaded from start are not extracted by default, but can be enabled with an extractor-arg. The extractor-arg `include_live_dash` is renamed to `include_incomplete_formats` to account for this new use-case.

Closes #1564
Authored by: Lesmiscore, pukkandan
2 years ago
pukkandan a057779d5e
[cleanup] Minor fixes
Closes #5129, Closes #4982
2 years ago
pukkandan 7a32c70d13
[cleanup] Fix flake8 and minor refactor
Issues from ab029d7e92, 1fb53b946c
2 years ago
pukkandan 709ee21417
[extractor/youtube] Do not warn on duplicate chapters
Eg: vYbaM8w8yzw
2 years ago
pukkandan 1fb53b946c
[extractor/youtube:tab] Improve continuation items extraction 2 years ago
pukkandan 1dd18a8808
[extractor/YoutubeShortsAudioPivot] Support `source` URLs
`ytshortsap:` is no longer needed
2 years ago
pukkandan 0a5095fe8d
[extractor/youtube:tab] Support `reporthistory` page
Closes #4929
2 years ago
coletdjnz 0ca0f88121
[extractor/heise] Fix extractor (#5029)
Fixes https://github.com/yt-dlp/yt-dlp/issues/1520
Authored by: coletdjnz
2 years ago
coletdjnz 80eb0bd9b9
[extractor/youtube] Add support for Shorts audio pivot feed (#4932)
This feed shows Shorts using the audio of a given video. 

ytshortsap: prefix can be used as a shortcut until YouTube
implements an official view. 

Closes #4911
Authored by: coletdjnz
2 years ago
coletdjnz c26f9b991a
[extractor/youtube] Support changing extraction language (#4470)
Adds `--extractor-args youtube:lang=<supported lang code>` extractor arg to prefer translated fields (e.g. title and description) of that language, if available, for all YouTube extractors. See README or error message for list of supported language codes.

Closes https://github.com/yt-dlp/yt-dlp/issues/387

Authored by: coletdjnz
2 years ago
coletdjnz 3ffb2f5bea
[extractor/youtube] Fix video like count extraction
Support new combined button layout
Authored by: coletdjnz
2 years ago
pukkandan 17ffed1842
[docs] Improvements
* Move detailed installation instructions to https://github.com/yt-dlp/yt-dlp/wiki/Installation
* Link to wiki where applicable
* Fix some mistakes. Closes #4853, Closes #4855, Closes #4852
* Improve some error messages
2 years ago
pukkandan 7c6eb424d3
[extractor/youtube] Detect `lazy-load-for-videos` embeds
Closes #4812
2 years ago
pukkandan 05deb747bb
[jsinterp] Fix escape in regex 2 years ago
pukkandan b505e8517a
[extractor/youtube] Fallback regex for nsig code extraction 2 years ago
coletdjnz 1ff88b7aec
[extractor/youtube] Add `no-youtube-prefer-utc-upload-date` compat option (#4771)
This option reverts 992f9a730b and 17322130a9 to prefer the non-UTC upload date in microformats.

Authored by: coletdjnz, pukkandan
2 years ago
pukkandan da4db748fa
[utils] Add `deprecation_warning`
See https://github.com/yt-dlp/yt-dlp/pull/2173#issuecomment-1097021515
2 years ago
pukkandan d81ba7d491
[jsinterp, extractor/youtube] Minor fixes 2 years ago
pukkandan c4b2df872d
[jsinterp] Fix `_separate`
Ref: https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1231126941
2 years ago
Samantaz Fox 224b5a35f7
[extractor/youtube] Update iOS Innertube clients (#4792)
Authored by: SamantazFox
2 years ago
coletdjnz 50ac0e5416
[extractor/youtube] Use device-specific user agent (#4770)
Thwart latest fingerprinting attempt (see https://github.com/iv-org/invidious/issues/3230#issuecomment-1226887639)

Authored by: coletdjnz
2 years ago
pukkandan 5e01315aa1
[cache, extractor/youtube] Invalidate old cache 2 years ago
pukkandan 992dc6b486
[jsinterp] Implement timeout
Workaround for #4716
2 years ago
pukkandan b25cac650f
[extractor/youtube] Fix bug in format sorting 2 years ago
pukkandan 90a1df305b
[test] Fix test_youtube_signature 2 years ago
pukkandan a831c2ea90
[cleanup] Misc 2 years ago
pukkandan 25836db6be
[extractor/youtube] Add fallback to phantomjs
Related #4635
2 years ago
pukkandan 580ce00782
[youtube] Improve signature caching
and refactor related functions
2 years ago
pukkandan f6ca640b12
[jsinterp] Fix for youtube player 1f7d5369
Closes #4635 again
2 years ago
pukkandan 3ce2933693
[youtube] Fix error reporting of "Incomplete data"
Related: #4669
2 years ago
pukkandan 5c6d2ef9d1
[youtube] Improve format sorting for IOS formats
When no itag/resolution is available for reference, use the closest resolution
2 years ago
Lesmiscore 62b58c0936
[docs] Consistent use of `e.g.` (#4643)
Authored by: Lesmiscore
2 years ago
pukkandan 8f53dc44a0
[jsinterp] Handle new youtube signature functions
Closes #4635
2 years ago
pukkandan 7e798d725e
[extractor] Fix format sorting of `channels` 2 years ago
coletdjnz c7dcf0b31e
[extractor/youtube] Add `androidSdkVersion` parameter to Android Innertube clients
Required to prevent YouTube returning a bad player response in some cases.

See: https://github.com/yt-dlp/yt-dlp/pull/4593, https://github.com/TeamNewPipe/NewPipe/issues/8713, https://github.com/iv-org/invidious/issues/3230, https://github.com/Tyrrrz/YoutubeExplode/issues/647

Authored by: coletdjnz
2 years ago
pukkandan a416623436
[extractor/youtube] Extract more format info 2 years ago
coletdjnz a3e9642116
[extractor/youtube] Prevent redirect to unwanted videos (#4593)
Example: https://www.youtube.com/watch?v=aQvGIIdgFDM

Authored by: coletdjnz
2 years ago
coletdjnz a0c830f488
[extractor/youtube] Bump Innertube client versions
YouTube may be requiring new versions soon. See https://github.com/iv-org/invidious/issues/3230, https://github.com/TeamNewPipe/NewPipe/issues/8713

Authored by: coletdjnz
2 years ago
pukkandan be5c1ae862
Standardize retry mechanism (#1649)
* [utils] Create `RetryManager`
* Migrate all retries to use the manager
* [extractor] Add wrapper methods for convenience
* Standardize console messages for retries
* Add `--retry-sleep` for extractors
2 years ago
pukkandan bfd973ece3 [extractors] Use new framework for existing embeds (#4307)
`Brightcove` is difficult to migrate because it's subclasses may depend
on the signature of the current functions. So it is left as-is for now

Note: Tests have not been migrated
2 years ago
pukkandan f2e8dbcc00 [extractor, test] Basic framework for embed tests (#4307)
and split download tests so they can be more easily run in CI

Authored by: coletdjnz
2 years ago
lazypete365 e325a21a1f
[extractor/youtube] Add `live_status=post_live` (#4495)
Related: https://github.com/yt-dlp/yt-dlp/issues/1564
Authored by: lazypete365
2 years ago
pukkandan 31b532a1f2
[cleanup] Misc 2 years ago
pukkandan c646d76f67
[webvtt, extractor/youtube] Extract auto-subs from livestream VODs
Closes #4130

Authored by: pukkandan, fstirlitz
2 years ago
pukkandan 07b47084ba
[extractor/youtube] Parse translated subtitles only when requested
Closes #4274
2 years ago
pukkandan 693f060040
[youtube,twitch] Allow waiting for channels to become live
Closes #2597
2 years ago
ftk 45e8a04e48
[extractor/youtube] More metadata for storyboards (#4334)
Authored by: ftk
2 years ago
pukkandan a3fb1ca5ab
[extractor/youtube] Fix duration check for post-live manifestless mode 2 years ago
Andrew 12a1b2254d
[extractor/youtube, cleanup] Fix tests (#4293)
Authored by: sheerluck
2 years ago
pukkandan a3976e0760
Improve chapter sanitization 2 years ago
pukkandan ae61d108dd
[cleanup] Misc cleanup 2 years ago
crazymoose77756 962ffcf89c
[cleanup] Fix some typos (#4194)
Authored by: crazymoose77756
2 years ago
pukkandan 14f25df2b6
[compat] Remove deprecated functions from core code 2 years ago
pukkandan ac66811112
[compat] Remove more functions
Removing any more will require changes to a large number of extractors
2 years ago
Brett824 06cc8f103b
[extractor/youtube] Mark videos as fully watched (#4146)
* Also fixes videos appearing as shorts in watch history

Closes #2555
Authored by: Brett824
2 years ago
pukkandan 9809740ba5
[extractor, cleanup] Reduce direct use of `_downloader` 2 years ago
pukkandan 471d0367c7
[youtube:clips] Support downloading clips
Closes #2543
2 years ago
pukkandan a70635b8a1
[cleanup, utils] Don't use kwargs for `format_field` 2 years ago
pukkandan 0df111a371
[youtube] Extract `comment_count` from webpage
Closes #4091
2 years ago
coletdjnz 4ce05f5759
[extractor/youtube] Fix live chat for videos with content warning
Fixes #4051
Authored by: coletdjnz
2 years ago
pukkandan 56ba69e4c9
[cleanup] Misc fixes
Closes #4027
2 years ago
coletdev 2ae778b8fc
[extractor/youtube] Add `innertube_host` and `innertube_key` extractor args (#3916)
Allows user to override Innertube API host or key for all requests
Authored by: coletdjnz
2 years ago
MMM 36195c4461
[dash] Show fragment count with `--live-from-start` (#3493)
Authored by: flashdagger
2 years ago
coletdjnz 65141660ab
[extractor/youtube] Fix bug in b7c47b7438
Closes #3997

Authored by: coletdjnz
2 years ago
pukkandan b7c47b7438
[extractor] Add `_search_json`
All fetching of JSON objects should eventually be done with this function
but only `youtube` is being refactored for now
2 years ago
pukkandan 1890fc6389
[cleanup] Misc fixes
Cherry-picks from: #3498, #3947
Related: #3949, https://github.com/yt-dlp/yt-dlp/issues/1839#issuecomment-1140313836
Authored by: pukkandan, flashdagger, gamer191
2 years ago