Commit Graph

253 Commits (edfc7725b1f2b4f7838836ca0df613ec0e058cac)

Author SHA1 Message Date
pukkandan f9934b9614
[cleanup] Mark some compat variables for removal (#2173)
Authored by fstirlitz, pukkandan
2 years ago
felix cfb0511d82
[cleanup] Remove unused code paths (#2173)
Notes:

* `_windows_write_string`: Fixed in 3.6
  * https://bugs.python.org/issue1602
  * PEP: https://www.python.org/dev/peps/pep-0528

* Windows UTF-8 fix: Fixed in 3.3
  * https://bugs.python.org/issue13216

* `__loader__`: is always present in 3.3+
  * https://bugs.python.org/issue14646

* `workaround_optparse_bug9161`: Fixed in 2.7
  * https://bugs.python.org/issue9161

Authored by: fstirlitz
2 years ago
coletdev 61d3665d9d
[youtube] Fix uploader for collaborative playlists (#3332)
Authored by: coletdjnz
2 years ago
pukkandan a44ca5a470
[cleanup] Misc fixes
Closes https://github.com/yt-dlp/yt-dlp/pull/3213, Closes https://github.com/yt-dlp/yt-dlp/pull/3117

Related: https://github.com/yt-dlp/yt-dlp/issues/3146#issuecomment-1077323114, https://github.com/yt-dlp/yt-dlp/pull/3277#discussion_r841019671, a825ffbffa (commitcomment-68538986), https://github.com/yt-dlp/yt-dlp/issues/2360, 5fa3c9a88f (r70393519), 5fa3c9a88f (r70393254)
2 years ago
pukkandan 04f3fd2c89
[cleanup] Use `_html_extract_title` 2 years ago
Fam0r fbfde1c3e6
[elonet] Rewrite extractor (#3277)
Closes #2911
Authored by: Fam0r, pukkandan
2 years ago
pukkandan 90137ca4be
[utils] Add `filter_dict` 2 years ago
pukkandan ae6a1b9585
[docs] Minor improvements
Closes #3127, Closes #3081, Closes #3177
2 years ago
pukkandan 231025c463
Fix bug in 52efa4b312
Closes #3173
2 years ago
pukkandan 700ccbe3f1
[extractor] Allow control characters inside json
Closes #3174
2 years ago
pukkandan 52efa4b312
[extractor] Add `_perform_login` function (#2943)
* Adds new functions `_initialize_pre_login` and `_perform_login` as part of the extractor API
* Adds `ie.supports_login` to the public API
2 years ago
coletdjnz a825ffbffa
[extractor] Support merging subtitles with data
Authored-by: coletdjnz
2 years ago
pukkandan 08d30158ec
[cleanup, docs] Misc cleanup
Closes #2828, closes #2734, closes #2802, closes #2937
2 years ago
pukkandan 497d2fab6c
[utils] Better traceback for `ExtractorError` 2 years ago
Zenon Mousmoulas 27231526ae
[ant1newsgr] Add extractor (#1982)
Authored by: zmousm
2 years ago
pukkandan a10aa588b0
[FormatSort] Consider `acodec`=`ogg` as `vorbis` 2 years ago
pukkandan 7265a2190c
Fix doubling of `video_id` in `ExtractorError` 2 years ago
coletdev ded9f32667
[extractor] Support `--mark-watched` without `_NETRC_MACHINE` (#2939)
Authored by: coletdjnz
2 years ago
Lesmiscore (Naoya Ozaki) 77cc7c6e60
[nhk] Add support for NHK for School (#2850)
Authored by: Lesmiscore
2 years ago
pukkandan ffa89477ea
[extractor] Fix for manifests without period duration
Closes #2705
Authored by: dirkf, pukkandan
2 years ago
pukkandan 297e9952b6
[extractor] Allow `http_headers` to be specified for `thumbnails` 2 years ago
pukkandan 079a7cfc71
[downloader] Do not use aria2c for non-native `m3u8`
Closes #2718
2 years ago
pukkandan 455a15e2dc
[cleanup,docs] Minor fixes
Closes #2541, #2484
2 years ago
pukkandan 85553414ae
[generic] Allow further processing of json_ld URL
Closes #2578
2 years ago
pukkandan fc08bdd6ab
[extractor] Allow non-fatal `title` extraction 2 years ago
pukkandan 2edb38e8ca
[extractor] Extract video inside `Article` json_ld
Closes #2448
2 years ago
pukkandan f40ee5e9a0
[extractor] Add convinience function `_yes_playlist` 2 years ago
pukkandan 6970b6005e
[cleanup] Minor fixes
Closes #2334
2 years ago
coletdjnz 6c73052c0a
[youtube] Extract channel subscriber count (#2399)
Closes #2350
* Adds `channel_follower_count` field
Authored-by: coletdjnz
2 years ago
pukkandan 311b6615d8
[extractor] Improve `url_result` and related 2 years ago
pukkandan 135dfa2c7e
[extractor,cleanup] Use `_search_nextjs_data` 2 years ago
MinePlayersPE 426764371f
[iq.com] Add extractors (#2354)
Closes #704
Authored by: MinePlayersPE
2 years ago
teridon 8bcd404818
[digitalconcerthall] Add extractor (#1931)
Authored by: teridon
2 years ago
coletdjnz f0d785d3ed
[youtube:tab] Extract more playlist metadata (#2069)
* Add fields modified_date, modified_timestamp
* Add field playlist_count
* [youtube:tab] Extract view_count, playlist_count, modified_date

Authored by: coletdjnz, pukkandan
2 years ago
chris f522573787
[extractor] Extract chapters from JSON-LD (#2031)
Authored by: iw0nderhow, pukkandan
2 years ago
nixxo 7592749cbe
[extractor] Extract thumbnails from JSON-LD (#2195)
Authored by: nixxo
2 years ago
Felix S 4afa3ec4b6
[extractor] Detect more subtitle codecs in MPD manifests (#2174)
Authored by: fstirlitz
2 years ago
MinePlayersPE d43de6821c
[GameJolt] Add extractors (#2036)
Authored by: MinePlayersPE
2 years ago
siddharth 774a46c53d
[npr] Make SMIL extraction non-fatal (#2099)
Closes #1934
Authored by: r5d
2 years ago
pukkandan b69fd25c25
[cleanup] Misc cleanup
Closes #1942 #1976 #2020 #2058 #1984
2 years ago
The Hatsune Daishi adbc4ec4bb
[dash,youtube] Download live from start to end (#888)
* Add option `--live-from-start` to enable downloading live videos from start
* Add key `is_from_start` in formats to identify formats (of live videos) that downloads from start
* [dash] Create protocol `http_dash_segments_generator` that allows a function to be passed instead of fragments
* [fragment] Allow multiple live dash formats to download simultaneously
* [youtube] Implement fragment re-fetching for the live dash formats
* [youtube] Re-extract dash manifest every 5 hours (manifest expires in 6hrs)
* [postprocessor/ffmpeg] Add `FFmpegFixupDuplicateMoovPP` to fixup duplicated moov atoms

Known issue: Ctrl+C doesn't work on Windows when downloading multiple formats

Closes #1521
Authored by: nao20010128nao, pukkandan
2 years ago
Zenon Mousmoulas d5c3254889
[extractor] Support default implicit graph in JSON-LD (#1983)
Original PR: https://github.com/ytdl-org/youtube-dl/pull/30229

Per W3C JSON-LD v1.1 §4.9 (non-normative ref):

    When a JSON-LD document's top-level structure is a map that contains
    no other keys than @graph and optionally @context (properties that
    are not mapped to an IRI or a keyword are ignored), @graph is
    considered to express the otherwise implicit default graph.

Authored by: zmousm
2 years ago
pukkandan a8549f19e7
[tiktok] Fix 53dad39e30 2 years ago
pukkandan 39ca3b5c7f
[extractor] Standardize `_live_title` 2 years ago
pukkandan 19188702ef
[FormatSort] Prevent incorrect deprecation warning
Closes #1981
2 years ago
The Hatsune Daishi 66f4c04e50
[extractor] Add `_search_nuxt_data` (#1921)
Authored by: nao20010128nao
2 years ago
pukkandan d2b2fca53f
[extractor] Ignore errors in comment extraction when `-i` is given
Closes #1787
2 years ago
pukkandan 2c4aaaddc9
[lazy_extractors] Fix for search IEs
Closes #1851
2 years ago
pukkandan ee8dd27a73
[cleanup] Add deprecation warnings 3 years ago
pukkandan f304da8a29
[cleanup] Misc cleanup
Closes #1805, closes #1800
3 years ago
pukkandan f2ebc5c7be
Option `--wait-for-video` to wait for scheduled streams 3 years ago
pukkandan 8f122fa070
[extractor] Extract `average_rating` from JSON-LD
Eg: Crunchyroll
3 years ago
pukkandan 234416e4bf
[downloader/ffmpeg] Fix for direct videos inside mpd manifests
Closes #1751
3 years ago
nyuszika7h 6b993ca765
[hls] Better FairPlay DRM detection (#1661)
Authored by: nyuszika7h
3 years ago
pukkandan 9ac24e235e
[curiositystream] Add more metadata
Closes #1568
3 years ago
Lauren Liberda f98709af31
[extractor] Add `_search_nextjs_data` (#1386)
Authored by: selfisekai
3 years ago
pukkandan 34921b4345
[utils] Add `join_nonempty` 3 years ago
pukkandan aeb2a9ad27
[FormatSort] `eac3` is better than `ac3` 3 years ago
pukkandan 0930b11fda
[docs,cleanup] Improve docs and minor cleanup
Closes #1387, #1404, #1408, #1485, #1415, #1450, #1492
3 years ago
pukkandan 10beccc980
[FormatSort] Fix some fields' defaults
Closes #1479
3 years ago
pukkandan 0db3bae879
[extractor] Fix some errors being converted to `ExtractorError` 3 years ago
pukkandan 96565c7e55
[cleanup] Add keyword automatically to SearchIE descriptions
and some minor cleanup of docs
3 years ago
pukkandan ec11a9f4a2
[minicurses] Add more colors 3 years ago
pukkandan 49a57e70a9
[cleanup] misc 3 years ago
pukkandan 176f1866cb
Add HDR information to formats 3 years ago
pukkandan e820fbaa6f
Do not verify thumbnail URLs by default
Partially reverts cca80fe611 and 0ba692acc8

Unless `--check-formats` is specified, this causes yt-dlp to return incorrect thumbnail urls.
See https://github.com/yt-dlp/yt-dlp/issues/340#issuecomment-877909966, #402

But the overhead in general use is not worth it

Closes #694, #725
3 years ago
pukkandan ec3f6640c1
[crunchyroll] Add season to flat-playlist
Closes #1319
3 years ago
pukkandan a0c716bb61
[instagram] Show appropriate error when login is needed
Closes #1264
3 years ago
pukkandan 6993f78d1b
[extractor,utils] Detect more codecs/mimetypes
Fixes: https://github.com/ytdl-org/youtube-dl/issues/29943
3 years ago
pukkandan a2160aa45f
[extractor] Generalize `getcomments` implementation 3 years ago
pukkandan cc16383ff3
[extractor] Simplify search extractors 3 years ago
pukkandan ba10757412 [extractor] Detect `EXT-X-KEY` Apple FairPlay 3 years ago
pukkandan b5ae35ee6d
[cleanup] Misc cleanup 3 years ago
pukkandan 819e05319b
Improved progress reporting (See desc) (#1125)
* Separate `--console-title` and `--no-progress`
* Add option `--progress` to show progress-bar even in quiet mode
* Fix and refactor `minicurses`
* Use `minicurses` for all progress reporting
* Standardize use of terminal sequences and enable color support for windows 10
* Add option `--progress-template` to customize progress-bar and console-title
* Add postprocessor hooks and progress reporting

Closes: #906, #901, #1085, #1170
3 years ago
Felix S 9359f3d4f0
[extractor] Extract storyboards from SMIL manifests (#1128)
Authored by: fstirlitz
3 years ago
pukkandan e6f21b3d92
[docs,cleanup] Some minor refactoring and improve docs 3 years ago
ChillingPepper 3cf4b91dc5
[SovietsCloset] Add duration from m3u8 (#908)
Authored by: ChillingPepper
3 years ago
pukkandan 1813a6ccd4
[youtube] Fix `--mark-watched` with `--cookies-from-browser`
Closes #1019
3 years ago
pukkandan bd6f722de8
dump files should obey `--trim-filename` (#1043)
Authored by: sulyi
3 years ago
DigitalDJ f9cc0161e6
[extractor] Fix root-relative URLs in MPD (#1006)
Authored by: DigitalDJ
3 years ago
pukkandan 0001fcb586
Add option `--netrc-location`
Closes #792, #963
3 years ago
pukkandan 198e3a04c9
[FormatSort] Remove priority of `lang` 3 years ago
pukkandan 6251555f1c
[downloader/ffmpeg] Support for DASH manifests (experimental)
Closes #159
3 years ago
pukkandan 1151c4079a
[extractor] Show video id in error messages if possible 3 years ago
pukkandan 88acdbc269
[extractor] Better error message for DRM (#729)
Closes #636
3 years ago
pukkandan 3fb4e21b38
[lazy_extractors] Fix `suitable` and add flake8 test 3 years ago
pukkandan 5ad28e7ffd
[extractor] Common function `_match_valid_url` 3 years ago
pukkandan a8731fcc1d minor bugfixes
bugs due to be2fc5b212, e9f4ccd19e
3 years ago
pukkandan be2fc5b212
[extractor] Detect `sttp` as subtitles in MPD
Closes #656
Solution by: fstirlitz
3 years ago
pukkandan b868936cd6 [cleanup] Misc 3 years ago
pukkandan 28f436bad0 [extractor] Reset non-repeating warnings per video 3 years ago
MinePlayersPE 3dbb2a9dcb
[RCTIPlus] Support events and TV (#625)
Authored by: MinePlayersPE
3 years ago
LE 68f5867cf0
[CBS] Add fallback (#579)
Related: https://github.com/ytdl-org/youtube-dl/issues/29564
Authored-by: llacb47, pukkandan
3 years ago
pukkandan dbf5416a20 [cleanup] Refactor some code 3 years ago
pukkandan f5510afef0
[FormatSort] Fix bug for audio with unknown codec 3 years ago
pukkandan ae30b84072
Add field `live_status` 3 years ago
pukkandan 0ba692acc8
[youtube] Extract more thumbnails
* The thumbnail URLs are hard-coded and their actual existence is tested lazily
* Added option `--no-check-formats` to not test them

Closes #340, Related: #402, #337, https://github.com/ytdl-org/youtube-dl/issues/29049
3 years ago
Felix S da1c94ee45
[generic] Extract previously missed subtitles (#515)
* [generic] Extract subtitles in cases missed previously
* [common] Detect discarded subtitles in SMIL manifests
* [generic] Extract everything in the SMIL manifest

Authored by: fstirlitz
3 years ago
felix 198f7ea89e
[extractor] Allow extracting multiple groups in `_search_regex`
From #497, Authored by: fstirlitz
3 years ago
pukkandan 6606817a86
[utils] Add `variadic` 3 years ago
pukkandan 4bb6b02f93
Improve `extractor_args` parsing 3 years ago
pukkandan 46890374f7
[extractor] Minor improvements (See desc)
1. Allow removal of login hint - extractors can set their own login hint as part of `msg`
2. Cleanup `_merge_subtitles` signature
3 years ago
pukkandan 60755938b3
[extractor] Prevent unnecessary download of hls manifests
and refactor `hls_split_discontinuity` code
3 years ago
pukkandan 5d3a0e794b
Add `--extractor-args` to pass extractor-specific arguments 3 years ago
pukkandan e36d50c5dd
[websockets] Add `WebSocketFragmentFD` (#399)
Necessary for #392

Co-authored by: nao20010128nao, pukkandan
3 years ago
pukkandan d0fb4bd16f
[pornhub] Extract `cast`
Closes #406, https://github.com/ytdl-org/youtube-dl/pull/27384
3 years ago
felix cdb19aa4c2
[downloader/mhtml] Add new downloader (#343)
This downloader is intended to be used for streams that consist of a
timed sequence of stand-alone images, such as slideshows or thumbnail
streams

This can be used for implementing:

https://github.com/ytdl-org/youtube-dl/issues/4974#issue-58006762
https://github.com/ytdl-org/youtube-dl/issues/4540#issuecomment-69574231
https://github.com/ytdl-org/youtube-dl/pull/11185#issuecomment-335554239

https://github.com/ytdl-org/youtube-dl/issues/9868
https://github.com/ytdl-org/youtube-dl/pull/14951


Authored by: fstirlitz
3 years ago
pukkandan 8326b00aab
Allow `images` formats
Necessary for #343.

* They are identified by `vcodec=acodec='none'`
* These formats show as the worst in `-F`
* Any postprocessor that expects audio/video will be skipped
* `b*` and all related selectors will skip such formats
* This commit also does not add any selector for downloading such formats. They have to be explicitly requested by the `format_id`. Implementation of a selector is left for when #389 is resolved
3 years ago
pukkandan 3b1fe47d84
Release 2021.06.08 3 years ago
felix e2efe599aa
[common] Fix FourCC fallback when parsing ISM (#372)
In some DASH manifests, the FourCC attribute is actually present,
but empty.  We thus apply the same fallback to 'AACL' that we do
when the attribute is entirely absent.

Authored by: fstirlitz
3 years ago
pukkandan b0089e8992
[fancode] Add extractor (#316,#354)
Closes #269, #363

Authored by: rmsmachine
3 years ago
pukkandan adddc50cbf
[extractor] Functions to parse socket.io response as json
Authored by: pukkandan, llacb47
3 years ago
pukkandan 37a3bb66a7
[extractor] Allow `note=False` when extracting manifests 3 years ago
pukkandan 177877c544
[extractor] Always prefer native hls downloader by default
When the manifest is not downloadable by native downloader, it already is able to detect it and switch to `ffmpeg`. So there doesn't seem to be a reason anymore to use ffmpeg as the preferred downloader
3 years ago
pukkandan 3907333c5d
[extractor] Skip subtitles without URI in m3u8 manifests
Closes #339

Authored by: hheimbuerger
3 years ago
pukkandan 9d5d4d64f8
[youtube] Better message when login required 3 years ago
pukkandan 07cce701de
[cleanup] linter, code formatting and readme 3 years ago
pukkandan a06916d98e
[extractor] Add `write_debug` and `get_param` 3 years ago
pukkandan 2412044c90
Add field `name` for subtitles
Co-authored by: pukkandan, tpikonen

Based on: #310, https://github.com/ytdl-org/youtube-dl/pull/26112
3 years ago
pukkandan 0760b0a7e2
Standardize `write_debug` 3 years ago
pukkandan 3d89341b47
[common] bugfix for when `compat_opts` is not given 3 years ago
pukkandan fac988053f
Release 2021.05.11
* and some documentation improvements
3 years ago
pukkandan 53ed7066ab
Option `--compat-options` to revert some of yt-dlp's changes
* Deprecates `--list-formats-as-table`, `--list-formats-old`
3 years ago
pukkandan 3158150cb7
[utils] Add `network_exceptions` 3 years ago
The Hatsune Daishi 0fa9a1e236
[whowatch] Add extractor #292
closes #223

Authored by: nao20010128nao 
Modified from: 9e4a0e061a/youtube_dl/extractor/whowatch.py
3 years ago
Felix S 66a1b8643a [downloader/ism] Support muxing TTML subtitles 3 years ago
Felix S 4a2f19abbd [downloader/hls] Assemble single-file WebVTT subtitles from HLS segments 3 years ago
Felix S f6a1d69a87 [extractor/common] Extend _extract_akamai_formats to also extract subtitle tracks 3 years ago
Felix S fd76a14259 [extractor/common, downloader/ism] Extract SSTR subtitle tracks
_parse_ism_formats was extended into _parse_ism_formats_and_subtitles;
all direct users were updated, though _extract_ism_formats was left
as a compatibility wrapper.

The SSTR downloader was also modified in order to prepare for muxing
subtitle streams, although no support for any subtitle codecs was
added in this commit.
3 years ago
Felix S 171e59edd4 [extractor/common] Extract DASH subtitle tracks
_extract_mpd_formats and _parse_mpd_formats were extended into
_…_formats_and_subtitles; wrappers with old names are provided
for compatibility.
3 years ago
Felix S a0c3b2d5cf [extractor/common] Extract HLS subtitle tracks
_extract_m3u8_formats is renamed to _extract_m3u8_formats_and_subtitles
and extended to handle subtitle tracks instead of skipping them;
a wrapper with the old name is provided for compatibility.

_parse_m3u8_formats is likewise renamed and extended, but without adding
the compatibility wrapper; the test suite is adjusted to test the enhanced
method instead.
3 years ago
Felix S 19bb39202d [extractor/common] Generalise _merge_subtitles
This allows modifying a subtitles dictionary in-place.
3 years ago
pukkandan 6a04a74e8b
[FormatSort] Fix for when some formats have quality and others don't 3 years ago
pukkandan b7da73eb19
Add option `--ignore-no-formats-error`
* Ignores the "no video format" and similar errors
* Experimental - Some extractors may still throw these errors
3 years ago
pukkandan 6a39ee13f7
Fix inconsistent use of `report_warning` 3 years ago
pukkandan b31fdeedfd
[documentation] Improvements 3 years ago
pukkandan f7ad71607d
Update to ytdl-commit-4fb25ff
[maoritv] Add new extractor
4fb25ff5a3

Except:
[vimeo] improve extraction 3ae9c0f410
[youtube:tab] Pass innertube context... 1b0a13f33c
3 years ago
pukkandan f37468c41f
Fix default of `dynamic_mpd` 3 years ago
pukkandan 73d4343e39
Fix some `m3u8` not obeying `--allow-unplayable-formats` 3 years ago
pukkandan 39ed931e53
Update to ytdl-commit-654b4f4
[youtube] prioritize information from YoutubeIE for playlist entries
654b4f4ff2
3 years ago
pukkandan b050d210df
Fix some typos and linter 3 years ago
Matthew c224251aad
[youtube] Show if video is `private`, `unlisted` etc in new field `availability` (#188)
Closes: #185, https://github.com/ytdl-org/youtube-dl/issues/25631

Authored by: colethedj, pukkandan
3 years ago
pukkandan e167860ce7
[documentation] Add deprecated options and aliases in readme 3 years ago
pukkandan e4beae703d
[formatSort] Fix `quality` being ignored
Closes #172
3 years ago
pukkandan 10db0d2f57
Update to ytdl-commit-3be0980
3be098010f
3 years ago
Matthew a1c5d2ca64
[Youtube] Rewrite comment extraction (#167)
Closes #121

TODO:
* Add an option for the user to specify newest/popular and max number of comments
* Refactor the download code and generalize with TabIE
* Parse time_text to timestamp
3 years ago
pukkandan fc21af505c
Fix some videos downloading with m3u8 extension 3 years ago
shirt-dev b3b30a4bca
Fix HLS playlist downloading (#127)
Co-authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>
3 years ago
pukkandan 5ef7d9bdd8
Release 2021.03.01 3 years ago
pukkandan 277d6ff5f2
Extract comments only when needed #95 (Closes #94) 3 years ago