Commit Graph

300 Commits (4093eb1fcc29a0e2aea9adfcba479787d9ae0c0c)

Author SHA1 Message Date
Matthew e756f45ba0
Improve handling for overriding extractors with plugins (#5916)
* Extractors replaced with plugin extractors now show in debug output
* Better testcase handling
* Added documentation
Authored by: coletdjnz, pukkandan
2 years ago
pukkandan 0b5546c723
[extractor] Let `_extract_format` functions obey `--ignore-no-formats` 2 years ago
pukkandan 71df9b7fd5
[cleanup] Misc 2 years ago
pukkandan 9bcfe33be7
[utils] Make `ExtractorError` mutable 2 years ago
pukkandan 9f14daf22b
[extractor] Deprecate `_sort_formats` 2 years ago
pukkandan 784320c98c
Implement universal format sorting
Closes #5566
2 years ago
pukkandan d0d74b7197
[utils] Move format sorting code into `utils` 2 years ago
pukkandan 105bfd90f5
Add new field `aspect_ratio`
Closes #5402
2 years ago
pukkandan 6368e2e639
[cleanup] Misc
Closes #5541
2 years ago
pukkandan 171a31dbe8
[extractor] Add a way to distinguish IEs that returns only videos 2 years ago
pukkandan f7fc8d39e9
[extractor] Fix `fatal=False` for `_search_nuxt_data`
Closes #5423
2 years ago
Matthew 0cf643b234
[extractor/youtube] Differentiate between no and disabled comments (#5491)
`comments` and `comment_count` will be set to None, as opposed to 
an empty list and 0, respectively.

Fixes https://github.com/yt-dlp/yt-dlp/issues/5068

Authored by: coletdjnz, pukkandan
2 years ago
pukkandan 46d09f8707
[cleanup] Lint and misc cleanup 2 years ago
pukkandan 62b8dac490
[extractor] Improve `_generic_title` 2 years ago
m4tu4g c13a301a94
[extractor/zeenews] Add extractor (#5289)
Closes #4967 
Authored by: m4tu4g, pukkandan
2 years ago
pukkandan d5d1df8afd
[cleanup Misc
Closes #5162
2 years ago
pukkandan 5225df50cf
[extractor/youtube:tab] Let `approximate_date` return timestamp 2 years ago
pukkandan ade1fa70cb
[extractor/generic] Separate embed extraction into own function (#5176) 2 years ago
Simon Sawicki 1d55ebabc9
[extractor/common] Fix `json_ld` type checks (#5145)
Closes #5144, #5143
Authored by: Grub4K
2 years ago
Matthew 867c66ff97
[extractor/youtube] Extract concurrent view count for livestreams (#5152)
Adds new field `concurrent_view_count`
Closes https://github.com/yt-dlp/yt-dlp/issues/4843

Authored by: coletdjnz
2 years ago
pukkandan a057779d5e
[cleanup] Minor fixes
Closes #5129, Closes #4982
2 years ago
coletdjnz d3a3d7f0cc
[extractor/JWPlatform] Fix extractor (#5112)
Fix bitrate and filesize extraction and support embeds with unquoted urls.

Related: #5106 

Authored by: coletdjnz
2 years ago
pukkandan 8b7fb8b60d
[extractor] Make search_json able to parse lists
Now `contains_pattern` can be set to `\[.+\]`
2 years ago
nixxo 81b6102d20
[downloader/ism] Support ec-3 codec (#5004)
Closes #296
Authored by: nixxo
2 years ago
pukkandan 0f60ba6e65
[extractor] Improve json+ld extraction
Related #5035
2 years ago
pukkandan 2fa669f759
[docs] Misc improvements
Closes #4987, Closes #4906, Closes #4919, Closes #4977, Closes #4979
2 years ago
pukkandan 8ca48a1a54
[extractor] Fix `fatal=False` in `RetryManager` 2 years ago
Simon Sawicki 8817a80d3a
[cookies] Parse cookies leniently (#4780)
Closes #4776, #3778
Authored by: Grub4K
2 years ago
pukkandan 2314b4d89f
Allow plugin extractors to replace the built-in ones
This allows easier plugin chaining; e.g.
- https://gist.github.com/pukkandan/24f13ff1ed385c5a390c1d7bd130d8f7
- https://gist.github.com/pukkandan/fcf5ca1785c80f64e471f0ee14f990fb
2 years ago
pukkandan 0cb0fdbbfe
[extractor/common] Escape `%` in `representation_id` of m3u8
Closes #4877
2 years ago
pukkandan 17ffed1842
[docs] Improvements
* Move detailed installation instructions to https://github.com/yt-dlp/yt-dlp/wiki/Installation
* Link to wiki where applicable
* Fix some mistakes. Closes #4853, Closes #4855, Closes #4852
* Improve some error messages
2 years ago
pukkandan d2c8aadf79
[cleanup] Misc
Closes #4710, Closes #4754, Closes #4723
Authored by: pukkandan, MrRawes, DavidH-2022
2 years ago
pukkandan f2e9fa3ef7
[FormatSort] Fix `aext` for `--prefer-free-formats`
Closes #4735
2 years ago
Lesmiscore bfbecd1174
[extractor/newspicks] Add extractor (#4725)
Authored by: Lesmiscore
2 years ago
pukkandan da4db748fa
[utils] Add `deprecation_warning`
See https://github.com/yt-dlp/yt-dlp/pull/2173#issuecomment-1097021515
2 years ago
pukkandan fe7866d0ed
Add option `--use-extractors`
Deprecates `--force-generic-extractor`

Closes #3234, Closes #2044

Related: #4307, #1791
2 years ago
Lesmiscore 62b58c0936
[docs] Consistent use of `e.g.` (#4643)
Authored by: Lesmiscore
2 years ago
pukkandan 7e798d725e
[extractor] Fix format sorting of `channels` 2 years ago
pukkandan b8ed0f15d4
[extractor] Add field `audio_channels` 2 years ago
pukkandan 0647d9251f
Minor bugfixes 2 years ago
pukkandan be5c1ae862
Standardize retry mechanism (#1649)
* [utils] Create `RetryManager`
* Migrate all retries to use the manager
* [extractor] Add wrapper methods for convenience
* Standardize console messages for retries
* Add `--retry-sleep` for extractors
2 years ago
pukkandan bfd973ece3 [extractors] Use new framework for existing embeds (#4307)
`Brightcove` is difficult to migrate because it's subclasses may depend
on the signature of the current functions. So it is left as-is for now

Note: Tests have not been migrated
2 years ago
pukkandan 1e8fe57e5c [extractor] Support multiple archive ids for one video (#4307)
Closes #4352
2 years ago
pukkandan f2e8dbcc00 [extractor, test] Basic framework for embed tests (#4307)
and split download tests so they can be more easily run in CI

Authored by: coletdjnz
2 years ago
pukkandan 8f97a15d1c [extractor] Framework for embed detection (#4307) 2 years ago
lazypete365 e325a21a1f
[extractor/youtube] Add `live_status=post_live` (#4495)
Related: https://github.com/yt-dlp/yt-dlp/issues/1564
Authored by: lazypete365
2 years ago
pukkandan b79f9e302d
`--compat-option no-live-chat` should disable danmaku
Closes #4387
2 years ago
pukkandan 6929b41a21
Remove Python 3.6 support
Closes #3764
2 years ago
pukkandan c6e07cf1e1
[cleanup] Misc 2 years ago
pukkandan 6edf28081f
[extractor] Passthrough `errnote=False` to parsing 2 years ago
Tim Weber eb2333bce1
[extractor/StarTrek] Add extractor (#4191)
Authored by: scy
2 years ago
ftk 45e8a04e48
[extractor/youtube] More metadata for storyboards (#4334)
Authored by: ftk
2 years ago
pukkandan 47046464fa
[extractor] Fix empty `BaseURL` in MPD
Closes #4113
2 years ago
crazymoose77756 962ffcf89c
[cleanup] Fix some typos (#4194)
Authored by: crazymoose77756
2 years ago
pukkandan c043c24625
[extractor] Fix `_create_request` when headers is None
Closes #4164
2 years ago
pukkandan 14f25df2b6
[compat] Remove deprecated functions from core code 2 years ago
pukkandan 54007a45f1
[cleanup] Consistent style for file heads 2 years ago
pukkandan ac66811112
[compat] Remove more functions
Removing any more will require changes to a large number of extractors
2 years ago
pukkandan 9809740ba5
[extractor, cleanup] Reduce direct use of `_downloader` 2 years ago
pukkandan 0c36dc00d7
[extractor/npr] Implement e50c3500b4 differently
Closes #4141
2 years ago
pukkandan 3975b4d2e8
Allow extractors to specify section_start/end for clips 2 years ago
pukkandan 612f2be5d3
Bugfix for 7b2c3f47c6 2 years ago
pukkandan 7b2c3f47c6
[cleanup] Misc 2 years ago
pukkandan 8072ef2bbd
[extractor/BiliIntl] Fix metadata extraction
Closes #4116
2 years ago
coletdev f0bc6e2019
[extractor] Add `default` parameter to `_search_json` (#4057)
Authored by: pukkandan, coletdjnz
2 years ago
pukkandan a70635b8a1
[cleanup, utils] Don't use kwargs for `format_field` 2 years ago
christoph-heinrich e121e3cee7
[cleanup] Minor fixes (#4096)
Authored by: christoph-heinrich
2 years ago
Evan Spensley 145c5a83a8
[extractor/GoogleDrive] Add folder extractor (#4009)
Closes #3388
Authored by: evansp, pukkandan
2 years ago
pukkandan f3c0c77304
[extractor] Handle `json_ld` with multiple `@type`s
Closes: #4022
2 years ago
pukkandan 56ba69e4c9
[cleanup] Misc fixes
Closes #4027
2 years ago
Aurélien Grosdidier d05460e5fe
[extractor/FranceCulture] Fix extractor (#3874)
Closes #3742
Authored by: aurelg, pukkandan
2 years ago
pukkandan b7c47b7438
[extractor] Add `_search_json`
All fetching of JSON objects should eventually be done with this function
but only `youtube` is being refactored for now
2 years ago
Lesmiscore 0bea4fd807
[extractor/0000studio] Add extractors (#3959)
Authored by: Lesmiscore
2 years ago
pukkandan 1890fc6389
[cleanup] Misc fixes
Cherry-picks from: #3498, #3947
Related: #3949, https://github.com/yt-dlp/yt-dlp/issues/1839#issuecomment-1140313836
Authored by: pukkandan, flashdagger, gamer191
2 years ago
pukkandan c4910024f3
[extractor] Fix bug in 617f658b7e
While the function signature don't enforce it, some IEs that override
`_download_webpage_handle` assume all optional arguments to be keyword-only

Closes #3954
2 years ago
siddharth ravikumar e50c3500b4
[extractor/npr] Use stream url from json-ld (#3455)
Closes #1934
Authored by: r5d
2 years ago
pukkandan 09d02ea429
[extractor] Fix bug in f95b9dee45
Closes #3951
2 years ago
pukkandan f95b9dee45
[extractor] Add dev option `--load-pages` 2 years ago
pukkandan 617f658b7e
[extractor, cleanup] Refactor `_download_...` methods 2 years ago
Lesmiscore 222a230871
[extractor/common] Recognize `src` attribute from HTML5 media elements (#3899)
Authored by: Lesmiscore
2 years ago
coletdjnz ee27297f82
[extractor/youtube] Fix initial player response extraction
Authored by: pukkandan, coletdjnz
2 years ago
pukkandan 8a82af3511
[cleanup] Misc fixes and cleanup
Closes #3780, Closes #3853, Closes #3850
2 years ago
pukkandan c487cf0010
[cleanup] Misc 3 years ago
pukkandan 2762dbb17e
[compat] Add `functools.cached_property` 3 years ago
pukkandan 21633673c3
[cleanup] Minor fixes 3 years ago
pukkandan 2414649192
[cleanup] Misc cleanup 3 years ago
pukkandan 82d020804d
[extractor] Use classmethod/property where possible
and refactor lazy extractors accordingly.

This reduces the need to create extractor instances
3 years ago
pukkandan 8dcce6a89c
[extractor] Document netrc machines
Closes #3169
3 years ago
pukkandan d4736fdb43
Remove warning for videos with an empty title 3 years ago
pukkandan 3fe75fdc80
[cleanup] Misc fixes (see desc)
* Do not warn when fixup is skipped for existing file
* [fragment] Fix `--skip-unavailable-fragments` for HTTP Errors
* [utils] write_string: Fix bug in 59f943cd50
* [utils] parse_codecs: Subtitle codec is generally referred to as `scodec`. https://github.com/yt-dlp/yt-dlp/pull/2174#discussion_r790156048
* [docs] Remove note about permissions. Closes #3597
3 years ago
pukkandan 1d485a1a79
[cleanup] Misc fixes
Closes #3565, https://github.com/yt-dlp/yt-dlp/issues/3514#issuecomment-1105944364
3 years ago
pukkandan 0a5a191a2a
Improve `--clean-infojson`
It should not removes fields that may be needed for `--load-infojson`.
Eg: `_ffmpeg_args`, `_has_drm`
3 years ago
Elyse a076c1f97a
[extractor] Update `manifest_url`s after redirect (#3575)
Authored by: elyse0
3 years ago
Elyse 779da8e31b
[extractor] Update dash `manifest_url` after redirects (#3563)
Closes #2696 
Authored by: elyse0
3 years ago
felix 77f9033095
[compat] Split into sub-modules (#2173)
Authored by: fstirlitz, pukkandan
3 years ago
pukkandan f82711587c
[cleanup] Sort imports
Using https://github.com/PyCQA/isort

    isort -m VERTICAL_HANGING_INDENT --py 36 -l 80 --rr -n --tc .
3 years ago
pukkandan 86e5f3ed2e
[cleanup] Upgrade syntax
Using https://github.com/asottile/pyupgrade

1. `__future__` imports and `coding: utf-8` were removed
2. Files were rewritten with `pyupgrade --py36-plus --keep-percent-format`
3. f-strings were cherry-picked from `pyupgrade --py36-plus`

Extractors are left untouched (except removing header) to avoid unnecessary merge conflicts
3 years ago
pukkandan f9934b9614
[cleanup] Mark some compat variables for removal (#2173)
Authored by fstirlitz, pukkandan
3 years ago
felix cfb0511d82
[cleanup] Remove unused code paths (#2173)
Notes:

* `_windows_write_string`: Fixed in 3.6
  * https://bugs.python.org/issue1602
  * PEP: https://www.python.org/dev/peps/pep-0528

* Windows UTF-8 fix: Fixed in 3.3
  * https://bugs.python.org/issue13216

* `__loader__`: is always present in 3.3+
  * https://bugs.python.org/issue14646

* `workaround_optparse_bug9161`: Fixed in 2.7
  * https://bugs.python.org/issue9161

Authored by: fstirlitz
3 years ago
coletdev 61d3665d9d
[youtube] Fix uploader for collaborative playlists (#3332)
Authored by: coletdjnz
3 years ago