Commit Graph

197 Commits (1ac7f461845b3f9c0c3a2e6a1308bf82d3e8e55a)

Author SHA1 Message Date
pukkandan 0edb3e336c
Do not prevent download if locking is unsupported
Closes #3022

Failure to lock download-archive is still fatal.
This is consistent with youtube-dl's behavior
3 years ago
pukkandan ce0593ef61
[http] Fix #3215 3 years ago
pukkandan a44ca5a470
[cleanup] Misc fixes
Closes https://github.com/yt-dlp/yt-dlp/pull/3213, Closes https://github.com/yt-dlp/yt-dlp/pull/3117

Related: https://github.com/yt-dlp/yt-dlp/issues/3146#issuecomment-1077323114, https://github.com/yt-dlp/yt-dlp/pull/3277#discussion_r841019671, a825ffbffa (commitcomment-68538986), https://github.com/yt-dlp/yt-dlp/issues/2360, 5fa3c9a88f (r70393519), 5fa3c9a88f (r70393254)
3 years ago
pukkandan c4f60dd7cd
[utils] Add `try_call` 3 years ago
pukkandan 28787f16c6
[downloader] Fix invocation of `HttpieFD`
Closes #3154
3 years ago
coletdev af14914baa
Remove `Accept-Encoding` header from `std_headers` (#3153)
This should be set by each downloader to what it supports.
Fixes https://github.com/yt-dlp/yt-dlp/issues/3142
Authored-by: coletdjnz
3 years ago
coletdev 8a23db9519
[wget] Fix proxy (#3152)
Upstream PR: https://github.com/ytdl-org/youtube-dl/pull/29343
Authored-by: kikuyan, coletdjnz
3 years ago
Lesmiscore (Naoya Ozaki) be4685ab7b
[http] Reject broken range before request (#3079)
* And fix filesize estimate for byterange downloads

Closes #2001
Authored by: Lesmiscore, Jules-A, pukkandan
3 years ago
coletdev a2e77303e3
[downloader/http] Retry on more errors (#3065)
Closes #3056, #2071
Related: #3034, #2969
Authored-by: coletdjnz
3 years ago
Lesmiscore (Naoya Ozaki) d71fd41249
[fragment] Read downloaded fragments only when needed (#3069)
Authored by: Lesmiscore
3 years ago
coletdev b3edc8068e
[downloader/mhtml] Fix fragments with absolute urls (#3044)
Authored-by: coletdjnz
3 years ago
pukkandan 592b748582
[cleanup] Minor cleanup
Closes #3006
3 years ago
pukkandan cf4f42cb97
Protect stdout from unexpected progress and console-title
Closes #3023
3 years ago
pukkandan 08d30158ec
[cleanup, docs] Misc cleanup
Closes #2828, closes #2734, closes #2802, closes #2937
3 years ago
Emanuel Hoogeveen 45806d44a7
[downloader] Obey `--file-access-retries` when deleting/renaming (#2224)
Authored by: ehoogeveen-medweb
3 years ago
Lesmiscore (Naoya Ozaki) 93c8410d33
[downloader/fragment] Fix bugs around resuming with Range (#2901)
Authored by: Lesmiscore
3 years ago
Lesmiscore 195c22840c
[downloader/fragment] Ignore `FileNotFoundError` when downloading livestreams
when `--live-from-start` is used for YouTube and the live ends, request for the last segment prematurely ends (or 404, 403).
this is causing lack of the file and `FileNotFoundError`
lacking segment doesn't have any data, so it's safe to ignore
3 years ago
Lesmiscore f0734e1190
[downloader/fragment] Fix problem where multiple threads can share one iterator
which causes "ValueError: generator already executing" error

Closes #2881
3 years ago
Lesmiscore (Naoya Ozaki) 15dfb3929c
[fc2:live] Add extractor (#2418)
Authored by: Lesmiscore
3 years ago
Lesmiscore (Naoya Ozaki) a539f06570
[downloader/fragment] Improve `--live-from-start` for YouTube livestreams (#2870) 3 years ago
pukkandan dcd55f766d
[aria2c] Add `--http-accept-gzip=true`
Closes #1936, #1236
3 years ago
pukkandan 079a7cfc71
[downloader] Do not use aria2c for non-native `m3u8`
Closes #2718
3 years ago
pukkandan 1e5d87beee
[websocket] Make syntax error in `websockets` module non-fatal
Closes #2633
3 years ago
pukkandan 1d3586d0d5
[aes] Add unpad_pkcs7 3 years ago
pukkandan af6793f804
[downloader/ffmpeg] Handle unknown formats better 3 years ago
pukkandan d57576b9d9
[httpie] Fix `available` method
Closes #2330
3 years ago
pukkandan f44afb54ef
[aria2c] Don't show progress when `--no-progress` 3 years ago
pukkandan 185bf31070
[youtube] End `live-from-start` properly when stream ends with 403
Closes #2089
3 years ago
Emanuel Hoogeveen 205a0654c0
Add option `--file-access-retries` (#2066)
Closes #517
Authored by: ehoogeveen-medweb
3 years ago
pukkandan b69fd25c25
[cleanup] Misc cleanup
Closes #1942 #1976 #2020 #2058 #1984
3 years ago
pukkandan 4ac5b94807
[dash] Fix `--test`
Bug in adbc4ec4bb
3 years ago
pukkandan 4273cc776d
[dash] Fix aria2c dash downloads
Bug in adbc4ec4bb
3 years ago
The Hatsune Daishi adbc4ec4bb
[dash,youtube] Download live from start to end (#888)
* Add option `--live-from-start` to enable downloading live videos from start
* Add key `is_from_start` in formats to identify formats (of live videos) that downloads from start
* [dash] Create protocol `http_dash_segments_generator` that allows a function to be passed instead of fragments
* [fragment] Allow multiple live dash formats to download simultaneously
* [youtube] Implement fragment re-fetching for the live dash formats
* [youtube] Re-extract dash manifest every 5 hours (manifest expires in 6hrs)
* [postprocessor/ffmpeg] Add `FFmpegFixupDuplicateMoovPP` to fixup duplicated moov atoms

Known issue: Ctrl+C doesn't work on Windows when downloading multiple formats

Closes #1521
Authored by: nao20010128nao, pukkandan
3 years ago
pukkandan f304da8a29
[cleanup] Misc cleanup
Closes #1805, closes #1800
3 years ago
pukkandan 7578d77d8c
[downloader] Add colors to download progress 3 years ago
pukkandan 234416e4bf
[downloader/ffmpeg] Fix for direct videos inside mpd manifests
Closes #1751
3 years ago
pukkandan d76991ab07
Fix `--check-formats` for `mhtml`
Closes #1709
3 years ago
nyuszika7h 6b993ca765
[hls] Better FairPlay DRM detection (#1661)
Authored by: nyuszika7h
3 years ago
pukkandan 59a7a13ef9
[docs] Minor documentation improvements
Closes #1583, #1599
3 years ago
pukkandan a4211baff5
[cleanup] Minor cleanup 3 years ago
pukkandan 8913ef74d7
[ffmpeg] Detect libavformat version for `aac_adtstoasc`
and print available features in verbose head
Based on https://github.com/ytdl-org/youtube-dl/pull/29581
3 years ago
pukkandan bd93fd5d45
[fragment] Fix progress display in fragmented downloads
Closes #1517
3 years ago
pukkandan 49a57e70a9
[cleanup] misc 3 years ago
pukkandan d3c93ec2b7
Don't create console for subprocesses on Windows (#1261)
Closes #1251
3 years ago
pukkandan b4b855ebc7
[fragment] Print error message when skipping fragment 3 years ago
pukkandan 2cda6b401d
Revert "[fragments] Pad fragments before decrypting (#1298)"
This reverts commit 373475f035.
3 years ago
pukkandan aa7785f860
[utils] Standardize timestamp formatting code
Closes #1285
3 years ago
pukkandan 9fab498fbf
[http] Retry on socket timeout
Closes #1222
3 years ago
shirt 373475f035
[fragments] Pad fragments before decrypting (#1298)
Closes #197, #1297, #1007
Authored by: shirt-dev
3 years ago
pukkandan 580d3274e5 [youtube] Expose different formats with same itag 3 years ago
pukkandan 03b4de722a
[downloader] Fix slow progress hooks
Closes #1301
3 years ago
pukkandan d5a39f0bad
[http] Show the last encountered error
Closes #1262
3 years ago
pukkandan fc5c8b6492
[eria2c] Fix --skip-unavailable fragment 3 years ago
pukkandan c111cefa5d
[downloader/ffmpeg] Improve simultaneous download and merge 3 years ago
pukkandan 21186af70a
[downloader] Fix throttledratelimit
The timer should not reset at start of each block
3 years ago
pukkandan d1d5c08f29
[minicurses] Fix when printing to file
Closes #1215
3 years ago
pukkandan b5ae35ee6d
[cleanup] Misc cleanup 3 years ago
pukkandan b922db9fe5
[http] Respect user-provided chunk size over extractor's 3 years ago
pukkandan 819e05319b
Improved progress reporting (See desc) (#1125)
* Separate `--console-title` and `--no-progress`
* Add option `--progress` to show progress-bar even in quiet mode
* Fix and refactor `minicurses`
* Use `minicurses` for all progress reporting
* Standardize use of terminal sequences and enable color support for windows 10
* Add option `--progress-template` to customize progress-bar and console-title
* Add postprocessor hooks and progress reporting

Closes: #906, #901, #1085, #1170
3 years ago
pukkandan 49e7e9c3ce
[docs,build] Change all pycryptodome references to pycryptodomex 3 years ago
shirt 7687c8ac6e
[HLS] Fix decryption issues (#1117)
* Unpad HLS fragments with PKCS#7 according to datatracker.ietf.org/doc/html/rfc8216
* media_sequence should only be incremented in for media fragments
* The native decryption should only be used if ffmpeg is unavailable since it is significantly slower. Closes #1086

Authored by: shirt-dev, pukkandan
3 years ago
coletdjnz 99e9e001de
[youtube] Cleanup authentication code (#786)
Authored by: coletdjnz
3 years ago
pukkandan be867b03f5
bugfix for bd50a52b0d 3 years ago
The Hatsune Daishi ee2b3563f3
[downloader/niconico] Pass custom headers (#1063)
Closes #1057
Authored by: nao20010128nao
3 years ago
The Hatsune Daishi bd50a52b0d
Basic framework for simultaneous download of multiple formats (#1036)
Authored by: nao20010128nao
3 years ago
pukkandan d806c9fd97
[docs,cleanup] Add deprecation warning in docs
for some counter intuitive behaviour that may be removed in future.

and fix linter
3 years ago
pukkandan 1009f67c2a
[fragment,aria2c] Generalize and refactor some code 3 years ago
pukkandan d9d8b85747
[fragment] Fix range header when using `-N` and media sequence (#1048)
Authored by: shirt
3 years ago
pukkandan daf7ac2b92
[fragment] Avoid repeated request for AES key 3 years ago
pukkandan 96933fc1b6
[aria2c] Fix IV for some AES-128 streams
Authored by: shirt
3 years ago
pukkandan 57aa7b8511
[hls] Byterange + AES128 is supported by native downloader 3 years ago
pukkandan edf65256aa
[hls,aes] Fallback to native implementation for AES-CBC
and detect `Cryptodome` in addition to `Crypto`

Closes #935
Related: #938
3 years ago
pukkandan d5fe04f5c7
Fix `--compat-option no-direct-merge` 3 years ago
Paul Wrubel d75201a873
Use `os.replace` where applicable (#793)
When using 
```py
os.remove(encodeFilename(filename))
os.rename(encodeFilename(temp_filename), encodeFilename(filename))
```
the `os.remove` need not be atomic and so can be executed arbitrarily compared to the immediately following rename call. It is better to use `os.replace` instead

Authored by: paulwrubel
3 years ago
pukkandan 691d5823d6
[aria2c] Obey `--rate-limit` 3 years ago
pukkandan 45842107b9
fix bug in 6251555f1c
:ci skip
3 years ago
pukkandan 6251555f1c
[downloader/ffmpeg] Support for DASH manifests (experimental)
Closes #159
3 years ago
pukkandan 330690a214
[downloader/ffmpeg] Allow passing custom arguments before -i
Closes #686
3 years ago
Felix S 7a6742b5f9
[webvtt] Fix timestamp overflow adjustment (#698)
In some streams, empty segments may appear with a bogus, non-monotone MPEG timestamp.
This should not be considered as an overflow

Authored by: fstirlitz
3 years ago
pukkandan f775c83110
Fix `--force-overwrites` when using `-k`
For formats that need merge, the `.fxxx` files are not removed before
downloading the corresponding `.part` files. This causes the rename to fail
3 years ago
Felix S 0715f7e19b
Revert erroneous use of the `Content-Length` header (#637)
This reverts commit 6c907eb33f

The use of the Content-Length value here is erroneous and may lead
to truncated downloads if a compression scheme is specified in the
Content-Encoding header, as the Content-Length header refers to the
size of encoded data, not of the raw bytestream. This has been noticed
in the wild with WebVTT subtitle segments.

Authored by: fstirlitz
3 years ago
Felix S 25a3f4f5d6
[webvtt] Merge daisy-chained duplicate cues (#638)
Fixes: https://github.com/yt-dlp/yt-dlp/issues/631#issuecomment-893338552

Previous deduplication algorithm only removed duplicate cues with
identical text, styles and timestamps.  This change also merges
cues that come in ‘daisy chains’, where sequences of cues with
identical text and styles appear in which the ending timestamp of
one equals the starting timestamp of the next.

This deduplication algorithm has the somewhat unfortunate side effect
that NOTE blocks between cues, if found, will be emitted in a different
order relative to their original cues.  This may be unwanted if perfect
fidelity is desired, but then so is daisy-chain deduplication itself.
NOTE blocks ought to be ignored by WebVTT players in any case.

Authored by: fstirlitz
3 years ago
pukkandan ad3dc496bb Misc fixes - See desc
* Remove unnecessary uses of _list_from_options_callback
* Fix download tests - Bug from 6e84b21559
* Rename ExecAfterDownloadPP to ExecPP and refactor its tests
* Ensure _write_ytdl_file closes file handle on error - Potential fix for #517
3 years ago
pukkandan 3448870205
[docs] Fix some mistakes and improve doc 3 years ago
pukkandan b868936cd6 [cleanup] Misc 3 years ago
pukkandan 9cc1a3130a
Fix resuming when using `--no-part`
Closes #576
3 years ago
nikhil bb36a55c41
[nbcolympics:stream] Fix extractor
PR: https://github.com/ytdl-org/youtube-dl/pull/29688
Closes: #617, https://github.com/ytdl-org/youtube-dl/issues/29665

* Livestreams are untested
* If using ffmpeg as downloader, v4.3+ is needed since `-http_seekable` option is necessary
* Instead of making a seperate key for each arg that needs to be passed to ffmpeg, I made `_ffmpeg_args`
* This deprecates `_seekable`, but the option is kept for compatibility

Authored by: nchilada, pukkandan
3 years ago
pukkandan e5611e8eda
[ffmpeg] Fix streaming `mp4` to `stdout` 3 years ago
pukkandan a46a815b05
[cleanup] Fix linter in 96fccc101f 3 years ago
pukkandan 96fccc101f [downloader] Allow streaming unmerged formats to stdout using ffmpeg
For this to work:
1. The downloader must be ffmpeg
2. The selected formats must have the same protocol
3. The formats must be downloadable by ffmpeg to stdout

Partial solution for: https://github.com/ytdl-org/youtube-dl/issues/28146, https://github.com/ytdl-org/youtube-dl/issues/27265
3 years ago
pukkandan dbf5416a20 [cleanup] Refactor some code 3 years ago
pukkandan f45e6c1126
[downloader] Pass same status object to all `progress_hooks` 3 years ago
pukkandan 3ba7740dd8
[downloader] Pass `info_dict` to `progress_hook`s 3 years ago
pukkandan 11f9be0912
[youtube] Extract data from multiple clients (#536)
* `player_client` accepts multiple clients
* default `player_client` = `android,web`
* music clients can be specifically requested
* Add IOS `player_client`
* Hide live dash since they can't be downloaded

Closes #501

Authored-by: pukkandan, colethedj
3 years ago
pukkandan 723d44b92b
[fragment] Handle errors in threads correctly 3 years ago
pukkandan 6b6c16ca6c [downloader/ffmpeg] Fix `--ppa` when using simultaneous download 3 years ago
siikamiika c2603313b1
[youtube_live_chat] use `clickTrackingParams` (#449)
Authored by: siikamiika
3 years ago
pukkandan bd4d1ea398
[cleanup] Minor refactoring of `fragment` 3 years ago
pukkandan 8e897ed283
[fragment] Return status of download correctly 3 years ago
siikamiika d534c4520b
[youtube_live_chat] Fix download with cookies (#437)
Closes #417 

Authored by: siikamiika
3 years ago