Commit Graph

308 Commits (315b3544296bb83012e20ee3af9d3cbf5600dd1c)

Author SHA1 Message Date
Simon Sawicki ff07792676
[core] Prevent RCE when using `--exec` with `%q` (CVE-2024-22423)
The shell escape function now properly escapes `%`, `\\` and `\n`. `utils.Popen` as well as `%q` output template expansion have been patched accordingly.

Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-hjq6-52gw-2g7p for more details.

Authored by: Grub4K
8 months ago
Simon Sawicki 979ce2e786
[test] `traversal`: Separate traversal tests (#9574)
Authored by: Grub4K
8 months ago
pukkandan 47ab66db0f
[docs] Misc Cleanup (#8977)
Closes #8355, #8944

Authored by: bashonly, Grub4k, Arthurszzz, seproDev, pukkandan

Co-authored-by: sepro <4618135+seproDev@users.noreply.github.com>
Co-authored-by: bashonly <bashonly@protonmail.com>
Co-authored-by: Arthurszzz <minecraftgamerarthur@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
Co-authored-by: bashonly <88596187+bashonly@users.noreply.github.com>
9 months ago
Simon Sawicki ffbd4f2a02
[utils] `traverse_obj`: Support `xml.etree.ElementTree.Element` (#8911)
Authored by: Grub4K
11 months ago
Simon Sawicki f9fb3ce86e
[cleanup] Misc (#8598)
Authored by: bashonly, pukkandan, seproDev, Grub4K

Co-authored-by: bashonly <bashonly@protonmail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: sepro <4618135+seproDev@users.noreply.github.com>
11 months ago
coletdjnz 196eb0fe77
[networking] Strip whitespace around header values (#8802)
Fixes https://github.com/yt-dlp/yt-dlp/issues/8729
Authored by: coletdjnz
12 months ago
Simon Sawicki 0b6f829b1d
[utils] `traverse_obj`: Move `is_user_input` into output template (#8673)
Authored by: Grub4K
12 months ago
Awal Garg 9d7ded6419
[utils] `js_to_json`: Fix `Date` constructor parsing (#8295)
Authored by: awalgarg, Grub4K
1 year ago
Simon Sawicki 088add9567
[cleanup] Misc
Authored by: Grub4K
1 year ago
Simon Sawicki de015e9307
[core] Prevent RCE when using `--exec` with `%q` (CVE-2023-40581)
The shell escape function is now using `""` instead of `\"`. `utils.Popen` has been patched to properly quote commands.

Prior to this fix using `--exec` together with `%q` when on Windows could cause remote code to execute. See https://github.com/yt-dlp/yt-dlp/security/advisories/GHSA-42h4-v29r-42qg for reference.

Authored by: Grub4K
1 year ago
bashonly 52414d64ca
[utils] `js_to_json`: Handle `Array` objects
Authored by: Grub4K, std-move

Co-authored-by: std-move <26625259+std-move@users.noreply.github.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>
1 year ago
pukkandan 3f7965105d
[utils] HTTPHeaderDict: Handle byte values 1 year ago
coletdjnz 4bf912282a
[networking] Remove dot segments during URL normalization (#7662)
This implements RFC3986 5.2.4 remove_dot_segments during the URL normalization process.

Closes #3355, #6526

Authored by: coletdjnz
1 year ago
bashonly af86873218
[utils] Improve `parse_duration`
Authored by: bashonly
1 year ago
coletdjnz 227bf1a33b
[networking] Rewrite architecture (#2861)
New networking interface consists of a `RequestDirector` that directs
each `Request` to appropriate `RequestHandler` and returns the
`Response` or raises `RequestError`. The handlers define adapters to
transform its internal Request/Response/Errors to our interfaces.

User-facing changes:
- Fix issues with per request proxies on redirects for urllib
- Support for `ALL_PROXY` environment variable for proxy setting
- Support for `socks5h` proxy
   - Closes https://github.com/yt-dlp/yt-dlp/issues/6325, https://github.com/ytdl-org/youtube-dl/issues/22618, https://github.com/ytdl-org/youtube-dl/pull/28093
- Raise error when using `https` proxy instead of silently converting it to `http`

Authored by: coletdjnz
1 year ago
pukkandan c365dba843
[networking] Add module (#2861)
No actual changes - code is only moved around
1 year ago
Mahmoud Abdel-Fattah 2af4eeb772
[utils] `clean_podcast_url`: Handle more trackers (#7556)
Authored by: mabdelfattah, bashonly
Closes #7544
1 year ago
pukkandan 4823ec9f46
Update to ytdl-commit-d1c6c5
[YouTube] [core] Improve platform debug log, based on yt-dlp
d1c6c5c4d6

Except:
    * 6ed34338285f722d0da312ce0af3a15a077a3e2a [jsinterp] Add short-cut evaluation for common expression
        * There was no performance improvement when tested with https://github.com/ytdl-org/youtube-dl/issues/30641
    * e8de54bce50f6f77a4d7e8e80675f7003d5bf630 [core] Handle `/../` sequences in HTTP URLs
        * We plan to implement this differently
2 years ago
Simon Sawicki b079c26f0a
[utils] `traverse_obj`: More fixes (#6959)
- Fix result when branching with `traverse_string`
- Fix `slice` path on `dict`s
- Fix tests and docstrings from 21b5ec86c2
- Add `is_iterable_like` helper function

Authored by: Grub4K
2 years ago
Simon Sawicki 21b5ec86c2
[utils] `traverse_obj`: Allow iterables in traversal (#6902)
Authored by: Grub4K
2 years ago
Simon Sawicki 0898c5c8cc
[utils] `js_to_json`: Implement template strings (#6623)
Authored by: Grub4K
2 years ago
Simon Sawicki 6839ae1f6d
[utils] `traverse_obj`: Fix more bugs
and cleanup uses of `default=[]`

Continued from b1bde57bef
2 years ago
Simon Sawicki b1bde57bef
[utils] `traverse_obj`: Fix several behavioral problems
See #6180 for further info

Authored by: Grub4K
2 years ago
Simon Sawicki 776995bc10
[utils] `traverse_obj`: Various improvements
- Add `set` key for transformations/filters
- Add `re.Match` group names
- Fix behavior for `expected_type` with `dict` key
- Raise for filter function signature mismatch in debug

Authored by: Grub4K
2 years ago
pukkandan fbb7383306
Add `weba` to known extensions 2 years ago
ChillingPepper d5f043d127
[utils] js_to_json: Fix bug in f55523c (#5771)
Authored by: ChillingPepper, pukkandan
2 years ago
Simon Sawicki a71b812f53
[utils] `js_to_json`: Improve escape handling (#5217)
Authored by: Grub4K
2 years ago
Matthew 4c9a1a3ba5
[extractor/wordpress:mb.miniAudioPlayer] Add embed extractor (#5087)
Closes https://github.com/yt-dlp/yt-dlp/issues/4994

Authored by: coletdjnz
2 years ago
Simon Sawicki 7b0127e1e1
[utils] `traverse_obj`: Allow `re.Match` objects (#5174)
Authored by: Grub4K
2 years ago
Simon Sawicki f99bbfc983
[utils] `traverse_obj`: Always return list when branching (#5170)
Fixes #5162
Authored by: Grub4K
2 years ago
Simon Sawicki ab029d7e92
[utils] `traverse_obj`: Rewrite, document and add tests (#5024)
Authored by: Grub4K
2 years ago
Elyse 7657ec7ed6
[utils] `base_url`: URL paths can contain `&` (#4841)
Authored by: elyse0
Closes #4187
2 years ago
pukkandan 8f53dc44a0
[jsinterp] Handle new youtube signature functions
Closes #4635
2 years ago
pukkandan 97d9c79e92
Fix tests for 989a01c261 2 years ago
Lauren N. Liberda fc61aff41b
Determine merge container better (See desc) (#1482)
* Determine the container early. Closes #4069
* Use codecs instead of just file extensions
* Obey `--prefer-free-formats`
* Allow fallbacks in `--merge-output`

Authored by: pukkandan, selfisekai
2 years ago
nixxo 47304e07dc
[extractor/rai] Add raisudtirol extractor (#4524)
Closes #4206
Authored by: nixxo
2 years ago
pukkandan 88f60feb32
Fix a904a7f8c6 2 years ago
Lesmiscore a904a7f8c6
Allow users to specify encoding in each config files (#4357)
Authored by: Lesmiscore
2 years ago
pukkandan f5ea47488a
[cleanup] Minor fixes 2 years ago
pukkandan 54007a45f1
[cleanup] Consistent style for file heads 2 years ago
pukkandan ac66811112
[compat] Remove more functions
Removing any more will require changes to a large number of extractors
2 years ago
pukkandan 0f06bcd759
[cleanup] Minor fixes (See desc)
* [youtube] Fix `--youtube-skip-dash-manifest`
* [build] Use `$()` in `Makefile`. Closes #3684
* Fix bug in 385ffb467b
* Fix bug in 43d7f5a5d0
* [cleanup] Remove unnecessary `utf-8` from `str.encode`/`bytes.decode`
* [utils] LazyList: Expose unnecessarily "protected" attributes
and other minor cleanup
3 years ago
pukkandan 19a0394044
[cleanup] Misc cleanup and refactor (#2173) 3 years ago
pukkandan f82711587c
[cleanup] Sort imports
Using https://github.com/PyCQA/isort

    isort -m VERTICAL_HANGING_INDENT --py 36 -l 80 --rr -n --tc .
3 years ago
pukkandan 86e5f3ed2e
[cleanup] Upgrade syntax
Using https://github.com/asottile/pyupgrade

1. `__future__` imports and `coding: utf-8` were removed
2. Files were rewritten with `pyupgrade --py36-plus --keep-percent-format`
3. f-strings were cherry-picked from `pyupgrade --py36-plus`

Extractors are left untouched (except removing header) to avoid unnecessary merge conflicts
3 years ago
felix cfb0511d82
[cleanup] Remove unused code paths (#2173)
Notes:

* `_windows_write_string`: Fixed in 3.6
  * https://bugs.python.org/issue1602
  * PEP: https://www.python.org/dev/peps/pep-0528

* Windows UTF-8 fix: Fixed in 3.3
  * https://bugs.python.org/issue13216

* `__loader__`: is always present in 3.3+
  * https://bugs.python.org/issue14646

* `workaround_optparse_bug9161`: Fixed in 2.7
  * https://bugs.python.org/issue9161

Authored by: fstirlitz
3 years ago
pukkandan b506289fe2
[test] Add `test_locked_file` 3 years ago
coletdev 1c1b2f96ae
[youtube:tab] Fix duration extraction for shorts (#3171)
Related: https://github.com/TeamNewPipe/NewPipe/issues/8034
Authored-by: coletdjnz
3 years ago
pukkandan 5c3895fff1
[outtmpl] Limit changes during sanitization
Closes #2761
3 years ago
s0u1h eeb2a770f3
[utils] `format_decimal_suffix`: Fix for very large numbers (#3109)
Authored by: s0u1h
3 years ago