The shell escape function now properly escapes `%`, `\\` and `\n`. `utils.Popen` as well as `%q` output template expansion have been patched accordingly.
Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-hjq6-52gw-2g7p for more details.
Authored by: Grub4K
Reverts 22e4dfacb6
Despite being documented as `Kbit/s`, the extractors/manifests were returning bitrates in SI units of kilobits/sec.
Authored by: seproDev, pukkandan
[build:Makefile] Restore compatibility with GNU Make <4.0
- The != variable assignment operator is not supported by GNU Make <4.0
- $(shell) is a no-op in BSD Make, assigns an empty string to the var
- Try to assign with != and fallback to $(shell) if not assigned (?=)
- Old versions of BSD find have different -exec behavior
- Pipe to `sed` instead of using `find ... -exec dirname {}`
- BSD tar does not support --transform, --owner or --group
- Allow user to specify path to GNU tar by passing GNUTAR variable
- pandoc vars are immediately evaluated with != in gmake>=4 and bmake
- Suppress stderr output for pandoc -v in case it is not installed
- Use string comparison instead of int comparison for pandoc version
Authored by: bashonly
- Only use trusted publishing with PyPI and remove support for PyPI tokens from release workflow
- Clean up improper actions syntax in the build workflow inputs
- Refactor Updater to allow for consistent unit testing with `UPDATE_SOURCES`
Authored by: bashonly
Closes https://github.com/yt-dlp/yt-dlp/issues/8206
Adds `raise_incomplete_data` extractor arg to revert this behaviour and raise an error.
Authored by: coletdjnz
Co-authored-by: Simon Sawicki <contact@grub4k.xyz>
The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.
Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.
Authored by: Grub4K
- Fixed support for IPv6 socks proxies
- Fixed support for IPv6 over socks5
- Fixed --source-address not being obeyed for socks4 and socks5
- Fixed socks4a when the destination address is an IPv4 address
Closes https://github.com/yt-dlp/yt-dlp/issues/7959
Fixes https://github.com/ytdl-org/youtube-dl/issues/15368
Authored by: coletdjnz
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
Co-authored-by: bashonly <bashonly@bashonly.com>
This also adds the following test runners:
- `3.12-dev` on `ubuntu-latest`
- `3.12-dev` on `windows-latest`
- `pypy-3.10` on `ubuntu-latest`
Authored by: Grub4K
Preference functions that take a request and a request handler instance can be registered to prioritize different request handlers per request.
Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
- Fix bug in `axel` Cookie header arg
- Pass cookies to `curl` as strings
- Write session cookies for `aria2c` and `wget`
Closes#7539
Authored by: bashonly
- Remove `Cookie` header from `http_headers` immediately after loading into cookiejar
- Restore compat for `--load-info-json` cookies
- Add more tests
- Fix improper passing of Cookie header by `MailRu` extractor
Closes#7558
Authored by: bashonly, pukkandan
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.
User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
- Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`
Authored by: coletdjnz
- ffmpeg: Calculate cookies from cookiejar and pass with `-cookies` arg instead of `-headers`
- aria2c, curl, wget: Write cookiejar to file and use external FD built-in cookiejar support
- httpie: Calculate cookies from cookiejar instead of `http_headers`
- axel: Calculate cookies from cookiejar and disable http redirection if cookies are passed
- May break redirects, but axel simply don't have proper cookie support
Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj
Authored by: bashonly, coletdjnz
This is useful for HLS where detecting whether the format is
actually DRM requires the child manifest to be downloaded.
Makes the error message when using `--test` inconsistent,
but doesn't really matter.
For both `--ignore-errors` and `--ignore-errors only_download`. Pass `--no-ignore-errors` to not ignore.
Closes https://github.com/yt-dlp/yt-dlp/issues/7474
Authored by: coletdjnz
* IOS is affected neither by 403 or by nsig so helps mitigate them preemptively
* IOS also has higher bit-rate "premium" formats though they are not labeled as such
Closes#1596
Previously, when some formats have accurate size and some approximate,
the ones with accurate size was always prioritized
For formats with known tbr and unknown vbr/abr, we were setting
(vbr=tbr, abr=0) for sorting to work. This is no longer needed.
Authored by pukkandan, u-spec-png
* Fixes `VrtNU` extractor to work with the VRT MAX site change
* Adapts `VRT`, `Ketnet` and `DagelijkseKost` extractors to the new VRT API
* Removes `Canvas` and `CanvasEen` extractors; the sites and API no longer exist
* Moves all remaining VRT-related extractors into the `vrt` module
Closes#4908
Authored by: jeroenj, bergoid, bashonly
Co-authored-by: bergoid <bergoid@users.noreply.github.com>
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
Adds new comment fields:
* `author_url` - The url to the comment author's page
* `author_is_verified` - Whether the author is verified on the platform
* `is_pinned` - Whether the comment is pinned to the top of the comments
Closes https://github.com/yt-dlp/yt-dlp/issues/5411
Authored by: coletdjnz
This adds `CrunchyrollMusicIE` and `CrunchyrollArtistIE` extractors using the new, reworked base class and expands the `CrunchyrollBetaIE` with support for movies and movie listings and more complete metadata extraction
Authored by: Grub4K
[YouTube] [core] Improve platform debug log, based on yt-dlp
d1c6c5c4d6
Except:
* 6ed34338285f722d0da312ce0af3a15a077a3e2a [jsinterp] Add short-cut evaluation for common expression
* There was no performance improvement when tested with https://github.com/ytdl-org/youtube-dl/issues/30641
* e8de54bce50f6f77a4d7e8e80675f7003d5bf630 [core] Handle `/../` sequences in HTTP URLs
* We plan to implement this differently
- Make single items collapse into one line
- Don't hide "Important changes" in `<details>`
- Move upstream merge into priority
- Properly support comma separated prefixes
Authored by: Grub4K
- Wait for build before publishing to PyPI
- Do not run `meta_files` job if release is cancelled
- Customizable channel in release workflow
- Display badges above changelog
Authored by: bashonly, Grub4K
- Fix result when branching with `traverse_string`
- Fix `slice` path on `dict`s
- Fix tests and docstrings from 21b5ec86c2
- Add `is_iterable_like` helper function
Authored by: Grub4K
Printing inside `finally` causes the order of logging to change
when there is an error, which is undesirable. So this is reverted.
The issue of `--print` being blocked by pre-processors was an
unintentional side-effect of changing the operation orders in
170605840e, and this is also partially
reverted.
* By default, stable will only update to stable, and nightly to nightly
Authored by: Grub4K, bashonly, pukkandan
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
- Add `set` key for transformations/filters
- Add `re.Match` group names
- Fix behavior for `expected_type` with `dict` key
- Raise for filter function signature mismatch in debug
Authored by: Grub4K
Avoids ending download early when live stream is temporarily offline.
Best used with somewhat large `--retry-sleep extractor:` and `--extractor-retries`
Authored by: mzhou
Use these to guard against future compat changes. This allows devs to
change defaults and make other potentially breaking changes more easily.
If you need everything to work exactly as-is, put this in your config
to make plugins easier to develop and use:
* Plugins are now loaded as namespace packages.
* Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.).
* Plugin packages can be installed and managed via pip, or dropped into any of the documented locations.
* Users do not need to edit any code files to install plugins.
* Backwards-compatible with previous plugin architecture.
As a side-effect, yt-dlp will now search in a few more locations for config files.
Closes https://github.com/yt-dlp/yt-dlp/issues/1389
Authored by: flashdagger, coletdjnz, pukkandan, Grub4K
Co-authored-by: Marcel <flashdagger@googlemail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
* Add support for VimeoPro URLs not containing a Vimeo video ID
* Add support for password-protected VimeoPro pages
Closes#5594
Authored by: bashonly, pukkandan
* `fragment_query`: passthrough any query in generic mpd/m3u8 manifest URLs to their fragments
* Add support for `extra_param_to_segment_url` to DASH downloader
Authored by: bashonly, pukkandan
* Move Brightcove embed extraction and tests into the IEs
* Split `BrightcoveNewBaseIE` from `BrightcoveNewIE`
* Fix bug in ade1fa70cb with the "wrong" spelling of `referrer` being smuggled
Closes#5539
`comments` and `comment_count` will be set to None, as opposed to
an empty list and 0, respectively.
Fixes https://github.com/yt-dlp/yt-dlp/issues/5068
Authored by: coletdjnz, pukkandan
* Check alternate domains when a URL does not work
* Obey `--no-check-formats`
* Remove webseeds (doesnt seem to exist anymore)
Authored by: flashdagger, pukkandan
Co-authored-by: Marcel <flashdagger@googlemail.com>
* The fragments are generated as a `LazyList`. So only the required formats are expanded during download, but all fragment lists are printed/written in infojson.
* The m3u8 formats which cannot be downloaded from start are not extracted by default, but can be enabled with an extractor-arg. The extractor-arg `include_live_dash` is renamed to `include_incomplete_formats` to account for this new use-case.
Closes#1564
Authored by: Lesmiscore, pukkandan
This feed shows Shorts using the audio of a given video.
ytshortsap: prefix can be used as a shortcut until YouTube
implements an official view.
Closes#4911
Authored by: coletdjnz
Adds `--extractor-args youtube:lang=<supported lang code>` extractor arg to prefer translated fields (e.g. title and description) of that language, if available, for all YouTube extractors. See README or error message for list of supported language codes.
Closes https://github.com/yt-dlp/yt-dlp/issues/387
Authored by: coletdjnz
description:Report issue with yt-dlp on a supported site
labels:[triage, site-bug]
labels:[triage, site-bug]
body:
body:
- type:checkboxes
- type:checkboxes
@ -7,7 +7,7 @@ body:
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
description:Fill all fields even if you think it is irrelevant for the issue
description:Fill all fields even if you think it is irrelevant for the issue
options:
options:
- label:I understand that I will be **blocked** if I remove or skip any mandatory\* field
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
required:true
- type:checkboxes
- type:checkboxes
id:checklist
id:checklist
@ -16,15 +16,15 @@ body:
description:|
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
options:
- label:I'm reporting a broken site
- label:I'm reporting that yt-dlp is broken on a **supported** site
required:true
required:true
- label:I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
required:true
@ -50,6 +50,8 @@ body:
options:
options:
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
required:true
required:true
- label:"If using API, add `'verbose': True` to `YoutubeDL` params instead"
required:false
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
description:Fill all fields even if you think it is irrelevant for the issue
description:Fill all fields even if you think it is irrelevant for the issue
options:
options:
- label:I understand that I will be **blocked** if I remove or skip any mandatory\* field
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
required:true
- type:checkboxes
- type:checkboxes
id:checklist
id:checklist
@ -18,13 +18,13 @@ body:
options:
options:
- label:I'm reporting a new site support request
- label:I'm reporting a new site support request
required:true
required:true
- label:I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
required:true
@ -62,6 +62,8 @@ body:
options:
options:
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
required:true
required:true
- label:"If using API, add `'verbose': True` to `YoutubeDL` params instead"
required:false
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
description:Fill all fields even if you think it is irrelevant for the issue
description:Fill all fields even if you think it is irrelevant for the issue
options:
options:
- label:I understand that I will be **blocked** if I remove or skip any mandatory\* field
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
required:true
- type:checkboxes
- type:checkboxes
id:checklist
id:checklist
@ -18,11 +18,11 @@ body:
options:
options:
- label:I'm requesting a site-specific feature
- label:I'm requesting a site-specific feature
required:true
required:true
- label:I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
required:true
@ -58,6 +58,8 @@ body:
options:
options:
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
required:true
required:true
- label:"If using API, add `'verbose': True` to `YoutubeDL` params instead"
required:false
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
description:Report a bug unrelated to any particular site or extractor
description:Report a bug unrelated to any particular site or extractor
labels:[triage, bug]
labels:[triage, bug]
body:
body:
@ -7,7 +7,7 @@ body:
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
description:Fill all fields even if you think it is irrelevant for the issue
description:Fill all fields even if you think it is irrelevant for the issue
options:
options:
- label:I understand that I will be **blocked** if I remove or skip any mandatory\* field
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
required:true
- type:checkboxes
- type:checkboxes
id:checklist
id:checklist
@ -18,13 +18,13 @@ body:
options:
options:
- label:I'm reporting a bug unrelated to a specific site
- label:I'm reporting a bug unrelated to a specific site
required:true
required:true
- label:I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
required:true
@ -43,6 +43,8 @@ body:
options:
options:
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
required:true
required:true
- label:"If using API, add `'verbose': True` to `YoutubeDL` params instead"
required:false
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
description:Fill all fields even if you think it is irrelevant for the issue
description:Fill all fields even if you think it is irrelevant for the issue
options:
options:
- label:I understand that I will be **blocked** if I remove or skip any mandatory\* field
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
required:true
- type:checkboxes
- type:checkboxes
id:checklist
id:checklist
@ -20,9 +20,9 @@ body:
required:true
required:true
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
required:true
- label:I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
required:true
@ -40,6 +40,8 @@ body:
label:Provide verbose output that clearly demonstrates the problem
label:Provide verbose output that clearly demonstrates the problem
options:
options:
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:"If using API, add `'verbose': True` to `YoutubeDL` params instead"
required:false
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
description:Fill all fields even if you think it is irrelevant for the issue
description:Fill all fields even if you think it is irrelevant for the issue
options:
options:
- label:I understand that I will be **blocked** if I remove or skip any mandatory\* field
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
required:true
- type:markdown
- type:markdown
attributes:
attributes:
@ -26,9 +26,9 @@ body:
required:true
required:true
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
required:true
- label:I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
required:true
@ -46,6 +46,8 @@ body:
label:Provide verbose output that clearly demonstrates the problem
label:Provide verbose output that clearly demonstrates the problem
options:
options:
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- label:"If using API, add `'verbose': True` to `YoutubeDL` params instead"
required:false
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
- label:Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below
description:Report issue with yt-dlp on a supported site
labels:[triage, site-bug]
labels:[triage, site-bug]
body:
body:
%(no_skip)s
%(no_skip)s
@ -10,15 +10,15 @@ body:
description:|
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
options:
- label:I'm reporting a broken site
- label:I'm reporting that yt-dlp is broken on a **supported** site
required:true
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've verified that I'm running yt-dlp version **%(version)s** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've verified that I'm running yt-dlp version **%(version)s** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
description:Report a bug unrelated to any particular site or extractor
description:Report a bug unrelated to any particular site or extractor
labels:[triage, bug]
labels:[triage, bug]
body:
body:
@ -12,13 +12,13 @@ body:
options:
options:
- label:I'm reporting a bug unrelated to a specific site
- label:I'm reporting a bug unrelated to a specific site
required:true
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
required:true
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
### Description of your *pull request* and other information
### Description of your *pull request* and other information
</details>
<!--
<!--
Explanation of your *pull request* in arbitrary form goes here. Please **make sure the description explains the purpose and effect** of your *pull request* and is worded well enough to be understood. Provide as much **context and examples** as possible
Explanation of your *pull request* in arbitrary form goes here. Please **make sure the description explains the purpose and effect** of your *pull request* and is worded well enough to be understood. Provide as much **context and examples** as possible
@ -32,7 +30,7 @@ Fixes #
- [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)
- [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8) and [ran relevant tests](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions)
### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)
@ -41,3 +39,5 @@ Fixes #
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] Core bug fix/improvement
- [ ] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))
# Use only 'java' to analyze code written in Java, Kotlin or both
# Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
steps:
- name:Checkout repository
uses:actions/checkout@v4
# Initializes the CodeQL tools for scanning.
- name:Initialize CodeQL
uses:github/codeql-action/init@v2
with:
languages:${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# Autobuild attempts to build any compiled languages (C/C++, C#, Go, Java, or Swift).
# If this step fails, then you should remove it and run the build manually (see below)
- name:Autobuild
uses:github/codeql-action/autobuild@v2
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
@ -79,7 +79,7 @@ Before reporting any issue, type `yt-dlp -U`. This should report that you're up-
### Is the issue already documented?
### Is the issue already documented?
Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/yt-dlp/yt-dlp/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2021.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/yt-dlp/yt-dlp/search?type=Issues) of this repository. If there is an issue, subscribe to it to be notified when there is any progress. Unless you have something useful to add to the conversation, please refrain from commenting.
Additionally, it is also helpful to see if the issue has already been documented in the [youtube-dl issue tracker](https://github.com/ytdl-org/youtube-dl/issues). If similar issues have already been reported in youtube-dl (but not in our issue tracker), links to them can be included in your issue report here.
Additionally, it is also helpful to see if the issue has already been documented in the [youtube-dl issue tracker](https://github.com/ytdl-org/youtube-dl/issues). If similar issues have already been reported in youtube-dl (but not in our issue tracker), links to them can be included in your issue report here.
@ -127,7 +127,7 @@ While these steps won't necessarily ensure that no misuse of the account takes p
### Is the website primarily used for piracy?
### Is the website primarily used for piracy?
We follow [youtube-dl's policy](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) to not support services that is primarily used for infringing copyright. Additionally, it has been decided to not to support porn sites that specialize in deep fake. We also cannot support any service that serves only [DRM protected content](https://en.wikipedia.org/wiki/Digital_rights_management).
We follow [youtube-dl's policy](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) to not support services that is primarily used for infringing copyright. Additionally, it has been decided to not to support porn sites that specialize in fakes. We also cannot support any service that serves only [DRM protected content](https://en.wikipedia.org/wiki/Digital_rights_management).
@ -138,14 +138,11 @@ Most users do not need to build yt-dlp and can [download the builds](https://git
To run yt-dlp as a developer, you don't need to build anything either. Simply execute
To run yt-dlp as a developer, you don't need to build anything either. Simply execute
python -m yt_dlp
python3 -m yt_dlp
To run the test, simply invoke your favorite test runner, or execute a test file directly; any of the following work:
To run all the available core tests, use:
python -m unittest discover
python3 devscripts/run_tests.py
python test/test_download.py
nosetests
pytest
See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
@ -154,14 +151,14 @@ If you want to create a build of yt-dlp yourself, you can follow the instruction
## Adding new feature or making overarching changes
## Adding new feature or making overarching changes
Before you start writing code for implementing a new feature, open an issue explaining your feature request and atleast one use case. This allows the maintainers to decide whether such a feature is desired for the project in the first place, and will provide an avenue to discuss some implementation details. If you open a pull request for a new feature without discussing with us first, do not be surprised when we ask for large changes to the code, or even reject it outright.
Before you start writing code for implementing a new feature, open an issue explaining your feature request and atleast one use case. This allows the maintainers to decide whether such a feature is desired for the project in the first place, and will provide an avenue to discuss some implementation details. If you open a pull request for a new feature without discussing with us first, do not be surprised when we ask for large changes to the code, or even reject it outright.
The same applies for changes to the documentation, code style, or overarching changes to the architecture
The same applies for changes to the documentation, code style, or overarching changes to the architecture
## Adding support for a new site
## Adding support for a new site
If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](https://www.github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. yt-dlp does **not support** such sites thus pull requests adding support for them **will be rejected**.
If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](#is-the-website-primarily-used-for-piracy)**. yt-dlp does **not support** such sites thus pull requests adding support for them **will be rejected**.
After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`):
After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`):
@ -187,15 +184,21 @@ After you have ensured this site is distributing its content legally, you can fo
'url': 'https://yourextractor.com/watch/42',
'url': 'https://yourextractor.com/watch/42',
'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
'info_dict': {
'info_dict': {
# For videos, only the 'id' and 'ext' fields are required to RUN the test:
'id': '42',
'id': '42',
'ext': 'mp4',
'ext': 'mp4',
'title': 'Video title goes here',
# Then if the test run fails, it will output the missing/incorrect fields.
'thumbnail': r're:^https?://.*\.jpg$',
# Properties can be added as:
# TODO more properties, either as:
# * A value, e.g.
# * A value
# 'title': 'Video title goes here',
# * MD5 checksum; start the string with md5:
# * MD5 checksum; start the string with 'md5:', e.g.
# * A regular expression; start the string with re:
# * A regular expression; start the string with 're:', e.g.
# 'thumbnail': r're:^https?://.*\.jpg$',
# * A count of elements in a list; start the string with 'count:', e.g.
# 'tags': 'count:10',
# * Any Python type, e.g.
# 'view_count': int,
}
}
}]
}]
@ -215,14 +218,14 @@ After you have ensured this site is distributing its content legally, you can fo
}
}
```
```
1. Add an import in [`yt_dlp/extractor/_extractors.py`](yt_dlp/extractor/_extractors.py). Note that the class name must end with `IE`.
1. Add an import in [`yt_dlp/extractor/_extractors.py`](yt_dlp/extractor/_extractors.py). Note that the class name must end with `IE`.
1. Run `python test/test_download.py TestDownload.test_YourExtractor` (note that `YourExtractor` doesn't end with `IE`). This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, the tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in. You can also run all the tests in one go with `TestDownload.test_YourExtractor_all`
1. Run `python3 devscripts/run_tests.py YourExtractor`. This *may fail* at first, but you can continually re-run it until you're done. Upon failure, it will output the missing fields and/or correct values which you can copy. If you decide to add more than one test, the tests will then be named `YourExtractor`, `YourExtractor_1`, `YourExtractor_2`, etc. Note that tests with an `only_matching` key in the test's dict are not included in the count. You can also run all the tests in one go with `YourExtractor_all`
1. Make sure you have atleast one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running.
1. Make sure you have atleast one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running.
1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L91-L426). Add tests and code for as many as you want.
1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L119-L440). Add tests and code for as many as you want.
1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
$ flake8 yt_dlp/extractor/yourextractor.py
$ flake8 yt_dlp/extractor/yourextractor.py
1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.7 and above. Backward compatibility is not required for even older versions of Python.
1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.8 and above. Backward compatibility is not required for even older versions of Python.
1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
$ git add yt_dlp/extractor/_extractors.py
$ git add yt_dlp/extractor/_extractors.py
@ -234,7 +237,7 @@ After you have ensured this site is distributing its content legally, you can fo
In any case, thank you very much for your contributions!
In any case, thank you very much for your contributions!
**Tip:** To test extractors that require login information, create a file `test/local_parameters.json` and add `"usenetrc": true` or your username and password in it:
**Tip:** To test extractors that require login information, create a file `test/local_parameters.json` and add `"usenetrc": true` or your `username`&`password` or `cookiefile`/`cookiesfrombrowser` in it:
```json
```json
{
{
"username": "your user name",
"username": "your user name",
@ -246,12 +249,12 @@ In any case, thank you very much for your contributions!
This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code.
This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code.
Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old yt-dlp versions working. Even though this breakage issue may be easily fixed by a new version of yt-dlp, this could take some time, during which the the extractor will remain broken.
Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old yt-dlp versions working. Even though this breakage issue may be easily fixed by a new version of yt-dlp, this could take some time, during which the extractor will remain broken.
### Mandatory and optional metafields
### Mandatory and optional metafields
For extraction to work yt-dlp relies on metadata your extractor extracts and provides to yt-dlp expressed by an [information dictionary](yt_dlp/extractor/common.py#L91-L426) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by yt-dlp:
For extraction to work yt-dlp relies on metadata your extractor extracts and provides to yt-dlp expressed by an [information dictionary](yt_dlp/extractor/common.py#L119-L440) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by yt-dlp:
- `id` (media identifier)
- `id` (media identifier)
- `title` (media title)
- `title` (media title)
@ -261,7 +264,7 @@ The aforementioned metafields are the critical data that the extraction does not
For pornographic sites, appropriate `age_limit` must also be returned.
For pornographic sites, appropriate `age_limit` must also be returned.
The extractor is allowed to return the info dict without url or formats in some special cases if it allows the user to extract usefull information with `--ignore-no-formats-error` - e.g. when the video is a live stream that has not started yet.
The extractor is allowed to return the info dict without url or formats in some special cases if it allows the user to extract useful information with `--ignore-no-formats-error` - e.g. when the video is a live stream that has not started yet.
[Any field](yt_dlp/extractor/common.py#219-L426) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
[Any field](yt_dlp/extractor/common.py#219-L426) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
@ -351,8 +354,9 @@ Say you extracted a list of thumbnails into `thumbnail_data` and want to iterate
```python
```python
thumbnail_data = data.get('thumbnails') or []
thumbnail_data = data.get('thumbnails') or []
thumbnails = [{
thumbnails = [{
'url': item['url']
'url': item['url'],
} for item in thumbnail_data] # correct
'height': item.get('h'),
} for item in thumbnail_data if item.get('url')] # correct
```
```
and not like:
and not like:
@ -360,12 +364,27 @@ and not like:
```python
```python
thumbnail_data = data.get('thumbnails')
thumbnail_data = data.get('thumbnails')
thumbnails = [{
thumbnails = [{
'url': item['url']
'url': item['url'],
'height': item.get('h'),
} for item in thumbnail_data] # incorrect
} for item in thumbnail_data] # incorrect
```
```
In this case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `or []` avoids this error and results in setting an empty list in `thumbnails` instead.
In this case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `or []` avoids this error and results in setting an empty list in `thumbnails` instead.
Alternately, this can be further simplified by using `traverse_obj`
```python
thumbnails = [{
'url': item['url'],
'height': item.get('h'),
} for item in traverse_obj(data, ('thumbnails', lambda _, v: v['url']))]
### Use convenience conversion and parsing functions
### Use convenience conversion and parsing functions
Wrap all extracted numeric data into safe functions from [`yt_dlp/utils.py`](yt_dlp/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
Wrap all extracted numeric data into safe functions from [`yt_dlp/utils/`](yt_dlp/utils/): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
Use `url_or_none` for safe URL processing.
Use `url_or_none` for safe URL processing.
@ -688,7 +707,7 @@ Use `traverse_obj` and `try_call` (superseeds `dict_get` and `try_get`) for safe
Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction.
Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction.
Explore [`yt_dlp/utils.py`](yt_dlp/utils.py) for more useful convenience functions.
Explore [`yt_dlp/utils/`](yt_dlp/utils/) for more useful convenience functions.
"short":"[priority] **A new release type has been added!**\n * [`nightly`](https://github.com/yt-dlp/yt-dlp/releases/tag/nightly) builds will be made after each push, containing the latest fixes (but also possibly bugs).\n * When using `--update`/`-U`, a release binary will only update to its current channel (either `stable` or `nightly`).\n * The `--update-to` option has been added allowing the user more control over program upgrades (or downgrades).\n * `--update-to` can change the release channel (`stable`, `nightly`) and also upgrade or downgrade to specific tags.\n * **Usage**: `--update-to CHANNEL`, `--update-to TAG`, `--update-to CHANNEL@TAG`"
"short":"[extractor/youtube] Add `ios` to default clients used\n - IOS is affected neither by 403 nor by nsig so helps mitigate them preemptively\n - IOS also has higher bit-rate 'premium' formats though they are not labeled as such",
"short":"[priority] Security: [[CVE-2023-35934](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35934)] Fix [Cookie leak](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj)\n - `--add-header Cookie:` is deprecated and auto-scoped to input URL domains\n - Cookies are scoped when passed to external downloaders\n - Add `cookies` field to info.json and deprecate `http_headers.Cookie`"
"short":"[priority] **The minimum *recommended* Python version has been raised to 3.8**\nSince Python 3.7 has reached end-of-life, support for it will be dropped soon. [Read more](https://github.com/yt-dlp/yt-dlp/issues/7803)"
"short":"[priority] Security: [[CVE-2023-40581](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40581)] [Prevent RCE when using `--exec` with `%q` on Windows](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg)\n - The shell escape function is now using `\"\"` instead of `\\\"`.\n - `utils.Popen` has been patched to properly quote commands."
"short":"[priority] **The release channels have been adjusted!**\n\t* [`master`](https://github.com/yt-dlp/yt-dlp-master-builds) builds are made after each push, containing the latest fixes (but also possibly bugs). This was previously the `nightly` channel.\n\t* [`nightly`](https://github.com/yt-dlp/yt-dlp-nightly-builds) builds are now made once a day, if there were any changes."
"short":"[priority] Security: [[CVE-2023-46121](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46121)] Patch [Generic Extractor MITM Vulnerability via Arbitrary Proxy Injection](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-3ch3-jhc6-5r8x)\n\t- Disallow smuggling of arbitrary `http_headers`; extractors now only use specific headers"
"short":"[priority] Security: [[CVE-2024-22423](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-22423)] [Prevent RCE when using `--exec` with `%q` on Windows](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-hjq6-52gw-2g7p)\n - The shell escape function now properly escapes `%`, `\\` and `\\n`.\n - `utils.Popen` has been patched accordingly."