The shell escape function now properly escapes `%`, `\\` and `\n`. `utils.Popen` as well as `%q` output template expansion have been patched accordingly.
Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-hjq6-52gw-2g7p for more details.
Authored by: Grub4K
Reverts 22e4dfacb6
Despite being documented as `Kbit/s`, the extractors/manifests were returning bitrates in SI units of kilobits/sec.
Authored by: seproDev, pukkandan
[build:Makefile] Restore compatibility with GNU Make <4.0
- The != variable assignment operator is not supported by GNU Make <4.0
- $(shell) is a no-op in BSD Make, assigns an empty string to the var
- Try to assign with != and fallback to $(shell) if not assigned (?=)
- Old versions of BSD find have different -exec behavior
- Pipe to `sed` instead of using `find ... -exec dirname {}`
- BSD tar does not support --transform, --owner or --group
- Allow user to specify path to GNU tar by passing GNUTAR variable
- pandoc vars are immediately evaluated with != in gmake>=4 and bmake
- Suppress stderr output for pandoc -v in case it is not installed
- Use string comparison instead of int comparison for pandoc version
Authored by: bashonly
- Only use trusted publishing with PyPI and remove support for PyPI tokens from release workflow
- Clean up improper actions syntax in the build workflow inputs
- Refactor Updater to allow for consistent unit testing with `UPDATE_SOURCES`
Authored by: bashonly
Closes https://github.com/yt-dlp/yt-dlp/issues/8206
Adds `raise_incomplete_data` extractor arg to revert this behaviour and raise an error.
Authored by: coletdjnz
Co-authored-by: Simon Sawicki <contact@grub4k.xyz>
The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.
Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.
Authored by: Grub4K
- Fixed support for IPv6 socks proxies
- Fixed support for IPv6 over socks5
- Fixed --source-address not being obeyed for socks4 and socks5
- Fixed socks4a when the destination address is an IPv4 address
Closes https://github.com/yt-dlp/yt-dlp/issues/7959
Fixes https://github.com/ytdl-org/youtube-dl/issues/15368
Authored by: coletdjnz
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
Co-authored-by: bashonly <bashonly@bashonly.com>
This also adds the following test runners:
- `3.12-dev` on `ubuntu-latest`
- `3.12-dev` on `windows-latest`
- `pypy-3.10` on `ubuntu-latest`
Authored by: Grub4K
Preference functions that take a request and a request handler instance can be registered to prioritize different request handlers per request.
Authored by: coletdjnz
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
- Fix bug in `axel` Cookie header arg
- Pass cookies to `curl` as strings
- Write session cookies for `aria2c` and `wget`
Closes#7539
Authored by: bashonly
- Remove `Cookie` header from `http_headers` immediately after loading into cookiejar
- Restore compat for `--load-info-json` cookies
- Add more tests
- Fix improper passing of Cookie header by `MailRu` extractor
Closes#7558
Authored by: bashonly, pukkandan
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.
User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
- Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`
Authored by: coletdjnz
- ffmpeg: Calculate cookies from cookiejar and pass with `-cookies` arg instead of `-headers`
- aria2c, curl, wget: Write cookiejar to file and use external FD built-in cookiejar support
- httpie: Calculate cookies from cookiejar instead of `http_headers`
- axel: Calculate cookies from cookiejar and disable http redirection if cookies are passed
- May break redirects, but axel simply don't have proper cookie support
Ref: https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj
Authored by: bashonly, coletdjnz
This is useful for HLS where detecting whether the format is
actually DRM requires the child manifest to be downloaded.
Makes the error message when using `--test` inconsistent,
but doesn't really matter.
For both `--ignore-errors` and `--ignore-errors only_download`. Pass `--no-ignore-errors` to not ignore.
Closes https://github.com/yt-dlp/yt-dlp/issues/7474
Authored by: coletdjnz
* IOS is affected neither by 403 or by nsig so helps mitigate them preemptively
* IOS also has higher bit-rate "premium" formats though they are not labeled as such
Closes#1596
Previously, when some formats have accurate size and some approximate,
the ones with accurate size was always prioritized
For formats with known tbr and unknown vbr/abr, we were setting
(vbr=tbr, abr=0) for sorting to work. This is no longer needed.
Authored by pukkandan, u-spec-png
* Fixes `VrtNU` extractor to work with the VRT MAX site change
* Adapts `VRT`, `Ketnet` and `DagelijkseKost` extractors to the new VRT API
* Removes `Canvas` and `CanvasEen` extractors; the sites and API no longer exist
* Moves all remaining VRT-related extractors into the `vrt` module
Closes#4908
Authored by: jeroenj, bergoid, bashonly
Co-authored-by: bergoid <bergoid@users.noreply.github.com>
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
Adds new comment fields:
* `author_url` - The url to the comment author's page
* `author_is_verified` - Whether the author is verified on the platform
* `is_pinned` - Whether the comment is pinned to the top of the comments
Closes https://github.com/yt-dlp/yt-dlp/issues/5411
Authored by: coletdjnz
This adds `CrunchyrollMusicIE` and `CrunchyrollArtistIE` extractors using the new, reworked base class and expands the `CrunchyrollBetaIE` with support for movies and movie listings and more complete metadata extraction
Authored by: Grub4K
[YouTube] [core] Improve platform debug log, based on yt-dlp
d1c6c5c4d6
Except:
* 6ed34338285f722d0da312ce0af3a15a077a3e2a [jsinterp] Add short-cut evaluation for common expression
* There was no performance improvement when tested with https://github.com/ytdl-org/youtube-dl/issues/30641
* e8de54bce50f6f77a4d7e8e80675f7003d5bf630 [core] Handle `/../` sequences in HTTP URLs
* We plan to implement this differently
- Make single items collapse into one line
- Don't hide "Important changes" in `<details>`
- Move upstream merge into priority
- Properly support comma separated prefixes
Authored by: Grub4K
- Wait for build before publishing to PyPI
- Do not run `meta_files` job if release is cancelled
- Customizable channel in release workflow
- Display badges above changelog
Authored by: bashonly, Grub4K
- Fix result when branching with `traverse_string`
- Fix `slice` path on `dict`s
- Fix tests and docstrings from 21b5ec86c2
- Add `is_iterable_like` helper function
Authored by: Grub4K
Printing inside `finally` causes the order of logging to change
when there is an error, which is undesirable. So this is reverted.
The issue of `--print` being blocked by pre-processors was an
unintentional side-effect of changing the operation orders in
170605840e, and this is also partially
reverted.
* By default, stable will only update to stable, and nightly to nightly
Authored by: Grub4K, bashonly, pukkandan
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
- Add `set` key for transformations/filters
- Add `re.Match` group names
- Fix behavior for `expected_type` with `dict` key
- Raise for filter function signature mismatch in debug
Authored by: Grub4K
Avoids ending download early when live stream is temporarily offline.
Best used with somewhat large `--retry-sleep extractor:` and `--extractor-retries`
Authored by: mzhou
Use these to guard against future compat changes. This allows devs to
change defaults and make other potentially breaking changes more easily.
If you need everything to work exactly as-is, put this in your config
to make plugins easier to develop and use:
* Plugins are now loaded as namespace packages.
* Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.).
* Plugin packages can be installed and managed via pip, or dropped into any of the documented locations.
* Users do not need to edit any code files to install plugins.
* Backwards-compatible with previous plugin architecture.
As a side-effect, yt-dlp will now search in a few more locations for config files.
Closes https://github.com/yt-dlp/yt-dlp/issues/1389
Authored by: flashdagger, coletdjnz, pukkandan, Grub4K
Co-authored-by: Marcel <flashdagger@googlemail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
* Add support for VimeoPro URLs not containing a Vimeo video ID
* Add support for password-protected VimeoPro pages
Closes#5594
Authored by: bashonly, pukkandan
* `fragment_query`: passthrough any query in generic mpd/m3u8 manifest URLs to their fragments
* Add support for `extra_param_to_segment_url` to DASH downloader
Authored by: bashonly, pukkandan
* Move Brightcove embed extraction and tests into the IEs
* Split `BrightcoveNewBaseIE` from `BrightcoveNewIE`
* Fix bug in ade1fa70cb with the "wrong" spelling of `referrer` being smuggled
Closes#5539
`comments` and `comment_count` will be set to None, as opposed to
an empty list and 0, respectively.
Fixes https://github.com/yt-dlp/yt-dlp/issues/5068
Authored by: coletdjnz, pukkandan
* Check alternate domains when a URL does not work
* Obey `--no-check-formats`
* Remove webseeds (doesnt seem to exist anymore)
Authored by: flashdagger, pukkandan
Co-authored-by: Marcel <flashdagger@googlemail.com>
* The fragments are generated as a `LazyList`. So only the required formats are expanded during download, but all fragment lists are printed/written in infojson.
* The m3u8 formats which cannot be downloaded from start are not extracted by default, but can be enabled with an extractor-arg. The extractor-arg `include_live_dash` is renamed to `include_incomplete_formats` to account for this new use-case.
Closes#1564
Authored by: Lesmiscore, pukkandan
This feed shows Shorts using the audio of a given video.
ytshortsap: prefix can be used as a shortcut until YouTube
implements an official view.
Closes#4911
Authored by: coletdjnz
Adds `--extractor-args youtube:lang=<supported lang code>` extractor arg to prefer translated fields (e.g. title and description) of that language, if available, for all YouTube extractors. See README or error message for list of supported language codes.
Closes https://github.com/yt-dlp/yt-dlp/issues/387
Authored by: coletdjnz
* [utils] Create `RetryManager`
* Migrate all retries to use the manager
* [extractor] Add wrapper methods for convenience
* Standardize console messages for retries
* Add `--retry-sleep` for extractors
`Brightcove` is difficult to migrate because it's subclasses may depend
on the signature of the current functions. So it is left as-is for now
Note: Tests have not been migrated
* Ignore private formats if not logged in (fixes https://github.com/yt-dlp/yt-dlp/issues/3832)
* Prefer original formats
* Support mpg formats
Authored by: coletdjnz, pukkandan
* Adds support for negative indices and step
* Add `-I` as alias for `--playlist-index`
* Deprecates `--playlist-start`, `--playlist-end`, `--playlist-reverse`, `--no-playlist-reverse`
Closes#2951, Closes#2853
While the function signature don't enforce it, some IEs that override
`_download_webpage_handle` assume all optional arguments to be keyword-only
Closes#3954
It is arguable how this "should" behave, but since progress is always
written to stdout in older yt-dl/p, we should keep it as-is
Bug in cf4f42cb97Closes#3844
* GHA does not cache python 3.6 for Windows, so use 3.8 instead
* Add tests for 3.11, PyPy3.8/3.9
* CPython 3.10 is now stable
* Do not pin Ubuntu to 18.04
* [youtube] Fix `--youtube-skip-dash-manifest`
* [build] Use `$()` in `Makefile`. Closes#3684
* Fix bug in 385ffb467b
* Fix bug in 43d7f5a5d0
* [cleanup] Remove unnecessary `utf-8` from `str.encode`/`bytes.decode`
* [utils] LazyList: Expose unnecessarily "protected" attributes
and other minor cleanup
* Do not warn when fixup is skipped for existing file
* [fragment] Fix `--skip-unavailable-fragments` for HTTP Errors
* [utils] write_string: Fix bug in 59f943cd50
* [utils] parse_codecs: Subtitle codec is generally referred to as `scodec`. https://github.com/yt-dlp/yt-dlp/pull/2174#discussion_r790156048
* [docs] Remove note about permissions. Closes#3597
On windows `conhost`, when `WINDOWS_VT_MODE` is enabled, `\n` is not
actually sent if the window is exactly the length of printed line,
and the line does not end with a white-space character. So the
line-break disappears when resizing the window.
Fixes#1863
Using https://github.com/asottile/pyupgrade
1. `__future__` imports and `coding: utf-8` were removed
2. Files were rewritten with `pyupgrade --py36-plus --keep-percent-format`
3. f-strings were cherry-picked from `pyupgrade --py36-plus`
Extractors are left untouched (except removing header) to avoid unnecessary merge conflicts
description:Report issue with yt-dlp on a supported site
labels:[triage, site-bug]
body:
- type:checkboxes
attributes:
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
description:Fill all fields even if you think it is irrelevant for the issue
options:
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
- type:checkboxes
id:checklist
attributes:
@ -9,15 +16,15 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm reporting a broken site
- label:I'm reporting that yt-dlp is broken on a **supported** site
required:true
- label:I've verified that I'm running yt-dlp version **2022.03.08.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
@ -26,37 +33,46 @@ body:
id:region
attributes:
label:Region
description:"Enter the region the site is accessible from"
placeholder:"India"
description:Enter the country/region that the site is accessible from
placeholder:India
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your issue in an arbitrary form.
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
label:Provide a description that is worded well enough to be understood
description:Fill all fields even if you think it is irrelevant for the issue
options:
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
- type:checkboxes
id:checklist
attributes:
@ -11,13 +18,13 @@ body:
options:
- label:I'm reporting a new site support request
required:true
- label:I've verified that I'm running yt-dlp version **2022.03.08.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
@ -26,8 +33,8 @@ body:
id:region
attributes:
label:Region
description:"Enter the region the site is accessible from"
placeholder:"India"
description:Enter the country/region that the site is accessible from
placeholder:India
- type:textarea
id:example-urls
attributes:
@ -43,31 +50,41 @@ body:
- type:textarea
id:description
attributes:
label:Description
description:|
Provide any additional information
placeholder:WRITE DESCRIPTION HERE
label:Provide a description that is worded well enough to be understood
description:Request a new functionality for a supported site
labels:[triage, site-enhancement]
body:
- type:checkboxes
attributes:
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
description:Fill all fields even if you think it is irrelevant for the issue
options:
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
- type:checkboxes
id:checklist
attributes:
@ -9,13 +16,13 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm reporting a site feature request
- label:I'm requesting a site-specific feature
required:true
- label:I've verified that I'm running yt-dlp version **2022.03.08.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
@ -24,8 +31,8 @@ body:
id:region
attributes:
label:Region
description:"Enter the region the site is accessible from"
placeholder:"India"
description:Enter the country/region that the site is accessible from
placeholder:India
- type:textarea
id:example-urls
attributes:
@ -39,33 +46,41 @@ body:
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your site feature request in an arbitrary form.
Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
label:Provide a description that is worded well enough to be understood
description:Report a bug unrelated to any particular site or extractor
labels:[triage, bug]
body:
- type:checkboxes
attributes:
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
description:Fill all fields even if you think it is irrelevant for the issue
options:
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
- type:checkboxes
id:checklist
attributes:
@ -11,46 +18,54 @@ body:
options:
- label:I'm reporting a bug unrelated to a specific site
required:true
- label:I've verified that I'm running yt-dlp version **2022.03.08.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your issue in an arbitrary form.
Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
label:Provide a description that is worded well enough to be understood
description:Request a new functionality unrelated to any particular site or extractor
labels:[triage, enhancement]
body:
- type:checkboxes
attributes:
label:DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
description:Fill all fields even if you think it is irrelevant for the issue
options:
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
- type:checkboxes
id:checklist
attributes:
@ -9,24 +16,51 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm reporting a feature request
- label:I'm requesting a feature unrelated to a specific site
required:true
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
- label:I've verified that I'm running yt-dlp version **2022.03.08.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your site feature request in an arbitrary form.
Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
label:Provide a description that is worded well enough to be understood
description:Fill all fields even if you think it is irrelevant for the issue
options:
- label:I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field
required:true
- type:markdown
attributes:
value:|
### Make sure you are **only** asking a question and not reporting a bug or requesting a feature.
If your question contains "isn't working" or "can you add", this is most likely the wrong template.
If you are in doubt whether this is the right template, **USE ANOTHER TEMPLATE**!
- type:checkboxes
id:checklist
attributes:
@ -9,45 +22,51 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm asking a question and **not** reporting a bug/feature request
- label:I'm asking a question and **not** reporting a bug or requesting a feature
required:true
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
- type:textarea
id:question
attributes:
label:Question
description:|
Ask your question in an arbitrary form.
Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information and as much context and examples as possible.
If your question contains "isn't working" or "can you add", this is most likely the wrong template.
If you are in doubt if this is the right template, use another template!
placeholder:WRITE QUESTION HERE
label:Please make sure the question is worded well enough to be understood
description:Report issue with yt-dlp on a supported site
labels:[triage, site-bug]
body:
%(no_skip)s
- type:checkboxes
id:checklist
attributes:
@ -9,15 +10,15 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm reporting a broken site
- label:I'm reporting that yt-dlp is broken on a **supported** site
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
@ -26,38 +27,14 @@ body:
id:region
attributes:
label:Region
description:"Enter the region the site is accessible from"
placeholder:"India"
description:Enter the country/region that the site is accessible from
placeholder:India
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your issue in an arbitrary form.
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
validations:
required:true
- type:textarea
id:log
attributes:
label:Verbose log
description:|
Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**.
Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below.
- label:I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
- label:I've checked that none of provided URLs [violate any copyrights](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
@ -26,8 +27,8 @@ body:
id:region
attributes:
label:Region
description:"Enter the region the site is accessible from"
placeholder:"India"
description:Enter the country/region that the site is accessible from
placeholder:India
- type:textarea
id:example-urls
attributes:
@ -43,32 +44,9 @@ body:
- type:textarea
id:description
attributes:
label:Description
description:|
Provide any additional information
placeholder:WRITE DESCRIPTION HERE
validations:
required:true
- type:textarea
id:log
attributes:
label:Verbose log
description:|
Provide the complete verbose output **using one of the example URLs provided above**.
Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below.
description:Request a new functionality for a supported site
labels:[triage, site-enhancement]
body:
%(no_skip)s
- type:checkboxes
id:checklist
attributes:
@ -9,13 +10,13 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm reporting a site feature request
- label:I'm requesting a site-specific feature
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
@ -24,8 +25,8 @@ body:
id:region
attributes:
label:Region
description:"Enter the region the site is accessible from"
placeholder:"India"
description:Enter the country/region that the site is accessible from
placeholder:India
- type:textarea
id:example-urls
attributes:
@ -39,34 +40,9 @@ body:
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your site feature request in an arbitrary form.
Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
validations:
required:true
- type:textarea
id:log
attributes:
label:Verbose log
description:|
Provide the complete verbose output of yt-dlp that demonstrates the need for the enhancement.
Add the `-vU` flag to your command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below.
description:Report a bug unrelated to any particular site or extractor
labels:[triage, bug]
body:
%(no_skip)s
- type:checkboxes
id:checklist
attributes:
@ -11,47 +12,22 @@ body:
options:
- label:I'm reporting a bug unrelated to a specific site
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've checked that all provided URLs are alive and playable in a browser
- label:I've checked that all provided URLs are playable in a browser with the same IP and same login details
required:true
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- label:I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your issue in an arbitrary form.
Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
validations:
required:true
- type:textarea
id:log
attributes:
label:Verbose log
description:|
Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**.
Add the `-vU` flag to **your** command line you run yt-dlp with (`yt-dlp -vU <your command line>`), copy the WHOLE output and insert it below.
description:Request a new functionality unrelated to any particular site or extractor
labels:[triage, enhancement]
body:
%(no_skip)s
- type:checkboxes
id:checklist
attributes:
@ -9,24 +10,22 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm reporting a feature request
- label:I'm requesting a feature unrelated to a specific site
required:true
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
- label:I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
- type:textarea
id:description
attributes:
label:Description
description:|
Provide an explanation of your site feature request in an arbitrary form.
Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information, any suggested solutions, and as much context and examples as possible
placeholder:WRITE DESCRIPTION HERE
label:Provide a description that is worded well enough to be understood
### Make sure you are **only** asking a question and not reporting a bug or requesting a feature.
If your question contains "isn't working" or "can you add", this is most likely the wrong template.
If you are in doubt whether this is the right template, **USE ANOTHER TEMPLATE**!
- type:checkboxes
id:checklist
attributes:
@ -9,45 +16,22 @@ body:
description:|
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
options:
- label:I'm asking a question and **not** reporting a bug/feature request
- label:I'm asking a question and **not** reporting a bug or requesting a feature
required:true
- label:I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- label:I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
required:true
- label:I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones
- label:I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions **including closed ones**. DO NOT post duplicates
required:true
- label:I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
required:true
- type:textarea
id:question
attributes:
label:Question
description:|
Ask your question in an arbitrary form.
Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
Provide any additional information and as much context and examples as possible.
If your question contains "isn't working" or "can you add", this is most likely the wrong template.
If you are in doubt if this is the right template, use another template!
placeholder:WRITE QUESTION HERE
label:Please make sure the question is worded well enough to be understood
**IMPORTANT**: PRs without the template will be CLOSED
### Description of your *pull request* and other information
<!--
Explanation of your *pull request* in arbitrary form goes here. Please **make sure the description explains the purpose and effect** of your *pull request* and is worded well enough to be understood. Provide as much **context and examples** as possible
-->
ADD DESCRIPTION HERE
Fixes #
<detailsopen><summary>Template</summary><!-- OPEN is intentional -->
<!--
# PLEASE FOLLOW THE GUIDE BELOW
- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes [ ] relevant to your *pull request* (like that [x])
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like
---
-->
### Before submitting a *pull request* make sure you have:
- [ ] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)
### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)
### What is the purpose of your *pull request*?
- [ ] Bug fix
- [ ] Improvement
- [ ] New extractor
- [ ] New feature
---
### Description of your *pull request* and other information
- [ ] Fix or improvement to an extractor (Make sure to add/update tests)
- [ ] New extractor ([Piracy websites will not be accepted](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#is-the-website-primarily-used-for-piracy))
- [ ] Core bug fix/improvement
- [ ] New feature (It is strongly [recommended to open an issue first](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-new-feature-or-making-overarching-changes))
Explanation of your *pull request* in arbitrary form goes here. Please make sure the description explains the purpose and effect of your *pull request* and is worded well enough to be understood. Provide as much context and examples as possible.
# Use only 'java' to analyze code written in Java, Kotlin or both
# Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
steps:
- name:Checkout repository
uses:actions/checkout@v4
# Initializes the CodeQL tools for scanning.
- name:Initialize CodeQL
uses:github/codeql-action/init@v2
with:
languages:${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
# queries: security-extended,security-and-quality
# Autobuild attempts to build any compiled languages (C/C++, C#, Go, Java, or Swift).
# If this step fails, then you should remove it and run the build manually (see below)
- name:Autobuild
uses:github/codeql-action/autobuild@v2
# ℹ️ Command-line programs to run using the OS shell.
# 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
# If the Autobuild fails above, remove it and uncomment the following three lines.
# modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.
@ -79,7 +79,7 @@ Before reporting any issue, type `yt-dlp -U`. This should report that you're up-
### Is the issue already documented?
Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/yt-dlp/yt-dlp/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2021.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/yt-dlp/yt-dlp/search?type=Issues) of this repository. If there is an issue, subscribe to it to be notified when there is any progress. Unless you have something useful to add to the conversation, please refrain from commenting.
Additionally, it is also helpful to see if the issue has already been documented in the [youtube-dl issue tracker](https://github.com/ytdl-org/youtube-dl/issues). If similar issues have already been reported in youtube-dl (but not in our issue tracker), links to them can be included in your issue report here.
@ -127,58 +127,93 @@ While these steps won't necessarily ensure that no misuse of the account takes p
### Is the website primarily used for piracy?
We follow [youtube-dl's policy](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) to not support services that is primarily used for infringing copyright. Additionally, it has been decided to not to support porn sites that specialize in deep fake. We also cannot support any service that serves only [DRM protected content](https://en.wikipedia.org/wiki/Digital_rights_management).
We follow [youtube-dl's policy](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) to not support services that is primarily used for infringing copyright. Additionally, it has been decided to not to support porn sites that specialize in fakes. We also cannot support any service that serves only [DRM protected content](https://en.wikipedia.org/wiki/Digital_rights_management).
# DEVELOPER INSTRUCTIONS
Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases) or get them via [the other installation methods](README.md#installation).
Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases), get them via [the other installation methods](README.md#installation) or directly run it using `python -m yt_dlp`.
To run yt-dlp as a developer, you don't need to build anything either. Simply execute
`yt-dlp` uses [`hatch`](<https://hatch.pypa.io>) as a project management tool.
You can easily install it using [`pipx`](<https://pipx.pypa.io>) via `pipx install hatch`, or else via `pip` or your package manager of choice. Make sure you are using at least version `1.10.0`, otherwise some functionality might not work as expected.
python -m yt_dlp
If you plan on contributing to `yt-dlp`, best practice is to start by running the following command:
To run the test, simply invoke your favorite test runner, or execute a test file directly; any of the following work:
```shell
$ hatch run setup
```
The above command will install a `pre-commit` hook so that required checks/fixes (linting, formatting) will run automatically before each commit. If any code needs to be linted or formatted, then the commit will be blocked and the necessary changes will be made; you should review all edits and re-commit the fixed version.
After this you can use `hatch shell` to enable a virtual environment that has `yt-dlp` and its development dependencies installed.
python -m unittest discover
python test/test_download.py
nosetests
pytest
In addition, the following script commands can be used to run simple tasks such as linting or testing (without having to run `hatch shell` first):
See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
While it is strongly recommended to use `hatch` for yt-dlp development, if you are unable to do so, alternatively you can manually create a virtual environment and use the following commands:
```shell
# To only install development dependencies:
$ python -m devscripts.install_deps --include dev
# Or, for an editable install plus dev dependencies:
$ python -m pip install -e ".[default,dev]"
# To setup the pre-commit hook:
$ pre-commit install
# To be used in place of `hatch test`:
$ python -m devscripts.run_tests
# To be used in place of `hatch fmt`:
$ ruff check --fix .
$ autopep8 --in-place .
# To only check code instead of applying fixes:
$ ruff check .
$ autopep8 --diff .
```
If you want to create a build of yt-dlp yourself, you can follow the instructions [here](README.md#compile).
## Adding new feature or making overarching changes
Before you start writing code for implementing a new feature, open an issue explaining your feature request and atleast one use case. This allows the maintainers to decide whether such a feature is desired for the project in the first place, and will provide an avenue to discuss some implementation details. If you open a pull request for a new feature without discussing with us first, do not be surprised when we ask for large changes to the code, or even reject it outright.
Before you start writing code for implementing a new feature, open an issue explaining your feature request and atleast one use case. This allows the maintainers to decide whether such a feature is desired for the project in the first place, and will provide an avenue to discuss some implementation details. If you open a pull request for a new feature without discussing with us first, do not be surprised when we ask for large changes to the code, or even reject it outright.
The same applies for changes to the documentation, code style, or overarching changes to the architecture
## Adding support for a new site
If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](https://www.github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. yt-dlp does **not support** such sites thus pull requests adding support for them **will be rejected**.
If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](#is-the-website-primarily-used-for-piracy)**. yt-dlp does **not support** such sites thus pull requests adding support for them **will be rejected**.
After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`):
1. [Fork this repository](https://github.com/yt-dlp/yt-dlp/fork)
# * A regular expression; start the string with 're:', e.g.
# 'thumbnail': r're:^https?://.*\.jpg$',
# * A count of elements in a list; start the string with 'count:', e.g.
# 'tags': 'count:10',
# * Any Python type, e.g.
# 'view_count': int,
}
}]
@ -215,27 +256,33 @@ After you have ensured this site is distributing its content legally, you can fo
# TODO more properties (see yt_dlp/extractor/common.py)
}
```
1. Add an import in [`yt_dlp/extractor/extractors.py`](yt_dlp/extractor/extractors.py).
1. Run `python test/test_download.py TestDownload.test_YourExtractor` (note that `YourExtractor` doesn't end with `IE`). This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, the tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in. You can also run all the tests in one go with `TestDownload.test_YourExtractor_all`
1. Make sure you have atleast one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running.
1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L91-L426). Add tests and code for as many as you want.
1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
1. Add an import in [`yt_dlp/extractor/_extractors.py`](yt_dlp/extractor/_extractors.py). Note that the class name must end with `IE`. Also note that when adding a parenthesized import group, the last import in the group must have a trailing comma in order for this formatting to be respected by our code formatter.
1. Run `hatch test YourExtractor`. This *may fail* at first, but you can continually re-run it until you're done. Upon failure, it will output the missing fields and/or correct values which you can copy. If you decide to add more than one test, the tests will then be named `YourExtractor`, `YourExtractor_1`, `YourExtractor_2`, etc. Note that tests with an `only_matching` key in the test's dict are not included in the count. You can also run all the tests in one go with `YourExtractor_all`
1. Make sure you have at least one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the particular test is disabled from running.
1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L119-L440). Add tests and code for as many as you want.
1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions), passes [ruff](https://docs.astral.sh/ruff/tutorial/#getting-started) code checks and is properly formatted:
```shell
$ hatch fmt --check
```
$ flake8 yt_dlp/extractor/yourextractor.py
You can use `hatch fmt` to automatically fix problems. Rules that the linter/formatter enforces should not be disabled with `# noqa` unless a maintainer requests it. The only exception allowed is for old/printf-style string formatting in GraphQL query templates (use `# noqa: UP031`).
1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.6 and above. Backward compatibility is not required for even older versions of Python.
1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.8 and above. Backward compatibility is not required for even older versions of Python.
1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
$ git add yt_dlp/extractor/extractors.py
$ git add yt_dlp/extractor/yourextractor.py
$ git commit -m '[yourextractor] Add extractor'
$ git push origin yourextractor
```shell
$ git add yt_dlp/extractor/_extractors.py
$ git add yt_dlp/extractor/yourextractor.py
$ git commit -m '[yourextractor] Add extractor'
$ git push origin yourextractor
```
1. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
In any case, thank you very much for your contributions!
**Tip:** To test extractors that require login information, create a file `test/local_parameters.json` and add `"usenetrc": true` or your username and password in it:
**Tip:** To test extractors that require login information, create a file `test/local_parameters.json` and add `"usenetrc": true` or your `username`&`password` or `cookiefile`/`cookiesfrombrowser` in it:
```json
{
"username": "your user name",
@ -247,12 +294,12 @@ In any case, thank you very much for your contributions!
This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code.
Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old yt-dlp versions working. Even though this breakage issue may be easily fixed by a new version of yt-dlp, this could take some time, during which the the extractor will remain broken.
Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old yt-dlp versions working. Even though this breakage issue may be easily fixed by a new version of yt-dlp, this could take some time, during which the extractor will remain broken.
### Mandatory and optional metafields
For extraction to work yt-dlp relies on metadata your extractor extracts and provides to yt-dlp expressed by an [information dictionary](yt_dlp/extractor/common.py#L91-L426) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by yt-dlp:
For extraction to work yt-dlp relies on metadata your extractor extracts and provides to yt-dlp expressed by an [information dictionary](yt_dlp/extractor/common.py#L119-L440) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by yt-dlp:
- `id` (media identifier)
- `title` (media title)
@ -262,7 +309,7 @@ The aforementioned metafields are the critical data that the extraction does not
For pornographic sites, appropriate `age_limit` must also be returned.
The extractor is allowed to return the info dict without url or formats in some special cases if it allows the user to extract usefull information with `--ignore-no-formats-error` - Eg: when the video is a live stream that has not started yet.
The extractor is allowed to return the info dict without url or formats in some special cases if it allows the user to extract useful information with `--ignore-no-formats-error` - e.g. when the video is a live stream that has not started yet.
[Any field](yt_dlp/extractor/common.py#219-L426) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
The latter will break extraction process with `KeyError` if `summary` disappears from `meta` at some later time but with the former approach extraction will just go ahead with `description` set to `None` which is perfectly fine (remember `None` is equivalent to the absence of data).
If the data is nested, do not use `.get` chains, but instead make use of the utility functions `try_get` or `traverse_obj`
If the data is nested, do not use `.get` chains, but instead make use of `traverse_obj`.
Considering the above `meta` again, assume you want to extract `["user"]["name"]` and put it in the resulting info dict as `uploader`
uploader = try_get(meta, lambda x: x['user']['name']) # old utility
```
Similarly, you should pass `fatal=False` when extracting optional data from a webpage with `_search_regex`, `_html_search_regex` or similar methods, for instance:
@ -347,26 +394,42 @@ On failure this code will silently continue the extraction with `description` se
Another thing to remember is not to try to iterate over `None`
Say you extracted a list of thumbnails into `thumbnail_data`using `try_get`and now want to iterate over them
Say you extracted a list of thumbnails into `thumbnail_data` and want to iterate over them
```python
thumbnail_data = try_get(...)
thumbnail_data = data.get('thumbnails') or []
thumbnails = [{
'url': item['url']
} for item in thumbnail_data or []] # correct
'url': item['url'],
'height': item.get('h'),
} for item in thumbnail_data if item.get('url')] # correct
```
and not like:
```python
thumbnail_data = try_get(...)
thumbnail_data = data.get('thumbnails')
thumbnails = [{
'url': item['url']
'url': item['url'],
'height': item.get('h'),
} for item in thumbnail_data] # incorrect
```
In the later case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `for item in thumbnail_data or []` avoids this error and results in setting an empty list in `thumbnails` instead.
In this case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `or []` avoids this error and results in setting an empty list in `thumbnails` instead.
Alternately, this can be further simplified by using `traverse_obj`
```python
thumbnails = [{
'url': item['url'],
'height': item.get('h'),
} for item in traverse_obj(data, ('thumbnails', lambda _, v: v['url']))]
@ -375,21 +438,21 @@ When extracting metadata try to do so from multiple sources. For example if `tit
#### Example
Say `meta` from the previous example has a `title` and you are about to extract it. Since `title` is a mandatory meta field you should end up with something like:
Say `meta` from the previous example has a `title` and you are about to extract it like:
```python
title = meta['title']
title = meta.get('title')
```
If `title` disappears from `meta` in future due to some changes on the hoster's side the extraction would fail since `title` is mandatory. That's expected.
If `title` disappears from `meta` in future due to some changes on the hoster's side the title extraction would fail.
Assume that you have some another source you can extract `title` from, for example `og:title` HTML meta of a `webpage`. In this case you can provide a fallback scenario:
Assume that you have some another source you can extract `title` from, for example `og:title` HTML meta of a `webpage`. In this case you can provide a fallback like:
```python
title = meta.get('title') or self._og_search_title(webpage)
```
This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`, making the extractor more robust.
### Regular expressions
@ -432,7 +495,7 @@ title = self._search_regex( # correct
which tolerates potential changes in the `style` attribute's value. Or even better:
```python
title = self._search_regex( # correct
@ -440,7 +503,7 @@ title = self._search_regex( # correct
webpage, 'title', group='title')
```
Note how you tolerate potential changes in the `style` attribute's value or switch from using double quotes to single for `class` attribute:
which also handles both single quotes in addition to double quotes.
The code definitely should not look like:
@ -458,7 +521,42 @@ title = self._search_regex( # incorrect
webpage, 'title', group='title')
```
Here the presence or absence of other attributes including `style` is irrelevent for the data we need, and so the regex must not depend on it
Here the presence or absence of other attributes including `style` is irrelevant for the data we need, and so the regex must not depend on it
#### Keep the regular expressions as simple as possible, but no simpler
Since many extractors deal with unstructured data provided by websites, we will often need to use very complex regular expressions. You should try to use the *simplest* regex that can accomplish what you want. In other words, each part of the regex must have a reason for existing. If you can take out a symbol and the functionality does not change, the symbol should not be there.
#### Do not misuse `.` and use the correct quantifiers (`+*?`)
Avoid creating regexes that over-match because of wrong use of quantifiers. Also try to avoid non-greedy matching (`?`) where possible since they could easily result in [catastrophic backtracking](https://www.regular-expressions.info/catastrophic.html)
Correct:
```python
title = self._search_regex(r'<span\b[^>]+class="title"[^>]*>([^<]+)', webpage, 'title')
```
Incorrect:
```python
title = self._search_regex(r'<span\b.*class="title".*>(.+?)<', webpage, 'title')
```
### Long lines policy
@ -467,7 +565,7 @@ There is a soft limit to keep lines of code under 100 characters long. This mean
For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit:
Conversely, don't unecessarily split small lines further. As a rule of thumb, if removing the line split keeps the code under 80 characters, it should be a single line.
Conversely, don't unnecessarily split small lines further. As a rule of thumb, if removing the line split keeps the code under 80 characters, it should be a single line.
Always use single quotes for strings (even if the string has `'`) and double quotes for docstrings. Use `'''` only for multi-line strings. An exception can be made if a string has multiple single quotes in it and escaping makes it significantly harder to read. For f-strings, use you can use double quotes on the inside. But avoid f-strings that have too many quotes inside.
Always use single quotes for strings (even if the string has `'`) and double quotes for docstrings. Use `'''` only for multi-line strings. An exception can be made if a string has multiple single quotes in it and escaping makes it *significantly* harder to read. For f-strings, use you can use double quotes on the inside. But avoid f-strings that have too many quotes inside.
### Inline values
Extracting variables is acceptable for reducing code duplication and improving readability of complex expressions. However, you should avoid extracting variables used only once and moving them to opposite parts of the extractor file, which makes reading the linear flow difficult.
#### Example
#### Examples
Correct:
```python
title = self._html_search_regex(r'<title>([^<]+)</title>', webpage, 'title')
title = self._html_search_regex(TITLE_RE, webpage, 'title')
# ...some lines of code...
return {
'title': title,
# ...some lines of code...
}
```
@ -574,33 +680,32 @@ Methods supporting list of patterns are: `_search_regex`, `_html_search_regex`,
### Trailing parentheses
Always move trailing parentheses used for grouping/functions after the last argument. On the other hand, literal list/tuple/dict/set should closed be in a new line. Generators and list/dict comprehensions may use either style
Always move trailing parentheses used for grouping/functions after the last argument. On the other hand, multi-line literal list/tuple/dict/set should closed be in a new line. Generators and list/dict comprehensions may use either style
### Use convenience conversion and parsing functions
Wrap all extracted numeric data into safe functions from [`yt_dlp/utils.py`](yt_dlp/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
Wrap all extracted numeric data into safe functions from [`yt_dlp/utils/`](yt_dlp/utils/): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
Use `url_or_none` for safe URL processing.
Use `try_get`, `dict_get` and `traverse_obj` for safe metadata extraction from parsed JSON.
Use `traverse_obj` and `try_call` (superseeds `dict_get` and `try_get`) for safe metadata extraction from parsed JSON.
Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction.
Explore [`yt_dlp/utils.py`](yt_dlp/utils.py) for more useful convenience functions.
Explore [`yt_dlp/utils/`](yt_dlp/utils/) for more useful convenience functions.
#### More examples
#### Examples
##### Safely extract optional description from parsed JSON
The `pending-fixes` label is added when there are changes requested to a PR. When the necessary changes are made, the label should be removed. However, despite our best efforts, it may sometimes happen that the maintainer did not see the changes or forgot to remove the label. If your PR is still marked as `pending-fixes` a few days after all requested changes have been made, feel free to ping the maintainer who labeled your issue and ask them to re-review and remove the label.
"short":"[priority] **A new release type has been added!**\n * [`nightly`](https://github.com/yt-dlp/yt-dlp/releases/tag/nightly) builds will be made after each push, containing the latest fixes (but also possibly bugs).\n * When using `--update`/`-U`, a release binary will only update to its current channel (either `stable` or `nightly`).\n * The `--update-to` option has been added allowing the user more control over program upgrades (or downgrades).\n * `--update-to` can change the release channel (`stable`, `nightly`) and also upgrade or downgrade to specific tags.\n * **Usage**: `--update-to CHANNEL`, `--update-to TAG`, `--update-to CHANNEL@TAG`"
"short":"[extractor/youtube] Add `ios` to default clients used\n - IOS is affected neither by 403 nor by nsig so helps mitigate them preemptively\n - IOS also has higher bit-rate 'premium' formats though they are not labeled as such",
"short":"[priority] Security: [[CVE-2023-35934](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-35934)] Fix [Cookie leak](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-v8mc-9377-rwjj)\n - `--add-header Cookie:` is deprecated and auto-scoped to input URL domains\n - Cookies are scoped when passed to external downloaders\n - Add `cookies` field to info.json and deprecate `http_headers.Cookie`"
"short":"[priority] **The minimum *recommended* Python version has been raised to 3.8**\nSince Python 3.7 has reached end-of-life, support for it will be dropped soon. [Read more](https://github.com/yt-dlp/yt-dlp/issues/7803)"
"short":"[priority] Security: [[CVE-2023-40581](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40581)] [Prevent RCE when using `--exec` with `%q` on Windows](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg)\n - The shell escape function is now using `\"\"` instead of `\\\"`.\n - `utils.Popen` has been patched to properly quote commands."
"short":"[priority] **The release channels have been adjusted!**\n\t* [`master`](https://github.com/yt-dlp/yt-dlp-master-builds) builds are made after each push, containing the latest fixes (but also possibly bugs). This was previously the `nightly` channel.\n\t* [`nightly`](https://github.com/yt-dlp/yt-dlp-nightly-builds) builds are now made once a day, if there were any changes."
"short":"[priority] Security: [[CVE-2023-46121](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-46121)] Patch [Generic Extractor MITM Vulnerability via Arbitrary Proxy Injection](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-3ch3-jhc6-5r8x)\n\t- Disallow smuggling of arbitrary `http_headers`; extractors now only use specific headers"
"short":"[priority] Security: [[CVE-2024-22423](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-22423)] [Prevent RCE when using `--exec` with `%q` on Windows](https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-hjq6-52gw-2g7p)\n - The shell escape function now properly escapes `%`, `\\` and `\\n`.\n - `utils.Popen` has been patched accordingly."
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)