[cleanup] Misc

This commit is contained in:
pukkandan 2022-11-30 11:34:51 +05:30
parent c9f5ce5118
commit 71df9b7fd5
No known key found for this signature in database
GPG key ID: 7EEE9E1E817D0A39
11 changed files with 72 additions and 40 deletions

View file

@ -12,13 +12,13 @@ jobs:
fail-fast: false fail-fast: false
matrix: matrix:
os: [ubuntu-latest] os: [ubuntu-latest]
# CPython 3.9 is in quick-test # CPython 3.11 is in quick-test
python-version: ['3.7', '3.10', 3.11-dev, pypy-3.7, pypy-3.8] python-version: ['3.8', '3.9', '3.10', pypy-3.7, pypy-3.8]
run-tests-ext: [sh] run-tests-ext: [sh]
include: include:
# atleast one of each CPython/PyPy tests must be in windows # atleast one of each CPython/PyPy tests must be in windows
- os: windows-latest - os: windows-latest
python-version: '3.8' python-version: '3.7'
run-tests-ext: bat run-tests-ext: bat
- os: windows-latest - os: windows-latest
python-version: pypy-3.9 python-version: pypy-3.9
@ -33,5 +33,6 @@ jobs:
run: pip install pytest run: pip install pytest
- name: Run tests - name: Run tests
continue-on-error: False continue-on-error: False
run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} core run: |
# Linter is in quick-test python3 -m yt_dlp -v || true # Print debug head
./devscripts/run_tests.${{ matrix.run-tests-ext }} core

View file

@ -10,24 +10,23 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
- name: Set up Python - name: Set up Python 3.11
uses: actions/setup-python@v4 uses: actions/setup-python@v4
with: with:
python-version: 3.9 python-version: '3.11'
- name: Install test requirements - name: Install test requirements
run: pip install pytest pycryptodomex run: pip install pytest pycryptodomex
- name: Run tests - name: Run tests
run: ./devscripts/run_tests.sh core run: |
python3 -m yt_dlp -v || true
./devscripts/run_tests.sh core
flake8: flake8:
name: Linter name: Linter
if: "!contains(github.event.head_commit.message, 'ci skip all')" if: "!contains(github.event.head_commit.message, 'ci skip all')"
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
- name: Set up Python - uses: actions/setup-python@v4
uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Install flake8 - name: Install flake8
run: pip install flake8 run: pip install flake8
- name: Make lazy extractors - name: Make lazy extractors

1
.gitignore vendored
View file

@ -71,6 +71,7 @@ dist/
zip/ zip/
tmp/ tmp/
venv/ venv/
.venv/
completions/ completions/
# Misc # Misc

View file

@ -351,8 +351,9 @@ #### Example
```python ```python
thumbnail_data = data.get('thumbnails') or [] thumbnail_data = data.get('thumbnails') or []
thumbnails = [{ thumbnails = [{
'url': item['url'] 'url': item['url'],
} for item in thumbnail_data] # correct 'height': item.get('h'),
} for item in thumbnail_data if item.get('url')] # correct
``` ```
and not like: and not like:
@ -360,12 +361,27 @@ #### Example
```python ```python
thumbnail_data = data.get('thumbnails') thumbnail_data = data.get('thumbnails')
thumbnails = [{ thumbnails = [{
'url': item['url'] 'url': item['url'],
'height': item.get('h'),
} for item in thumbnail_data] # incorrect } for item in thumbnail_data] # incorrect
``` ```
In this case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `or []` avoids this error and results in setting an empty list in `thumbnails` instead. In this case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `or []` avoids this error and results in setting an empty list in `thumbnails` instead.
Alternately, this can be further simplified by using `traverse_obj`
```python
thumbnails = [{
'url': item['url'],
'height': item.get('h'),
} for item in traverse_obj(data, ('thumbnails', lambda _, v: v['url']))]
```
or, even better,
```python
thumbnails = traverse_obj(data, ('thumbnails', ..., {'url': 'url', 'height': 'h'}))
```
### Provide fallbacks ### Provide fallbacks

View file

@ -432,19 +432,19 @@ ## Geo-restriction:
explicitly provided IP block in CIDR notation explicitly provided IP block in CIDR notation
## Video Selection: ## Video Selection:
-I, --playlist-items ITEM_SPEC Comma separated playlist_index of the videos -I, --playlist-items ITEM_SPEC Comma separated playlist_index of the items
to download. You can specify a range using to download. You can specify a range using
"[START]:[STOP][:STEP]". For backward "[START]:[STOP][:STEP]". For backward
compatibility, START-STOP is also supported. compatibility, START-STOP is also supported.
Use negative indices to count from the right Use negative indices to count from the right
and negative STEP to download in reverse and negative STEP to download in reverse
order. E.g. "-I 1:3,7,-5::2" used on a order. E.g. "-I 1:3,7,-5::2" used on a
playlist of size 15 will download the videos playlist of size 15 will download the items
at index 1,2,3,7,11,13,15 at index 1,2,3,7,11,13,15
--min-filesize SIZE Do not download any videos smaller than --min-filesize SIZE Abort download if filesize is smaller than
SIZE, e.g. 50k or 44.6M
--max-filesize SIZE Abort download if filesize is larger than
SIZE, e.g. 50k or 44.6M SIZE, e.g. 50k or 44.6M
--max-filesize SIZE Do not download any videos larger than SIZE,
e.g. 50k or 44.6M
--date DATE Download only videos uploaded on this date. --date DATE Download only videos uploaded on this date.
The date can be "YYYYMMDD" or in the format The date can be "YYYYMMDD" or in the format
[now|today|yesterday][-N[day|week|month|year]]. [now|today|yesterday][-N[day|week|month|year]].
@ -491,9 +491,9 @@ ## Video Selection:
a file that is in the archive a file that is in the archive
--break-on-reject Stop the download process when encountering --break-on-reject Stop the download process when encountering
a file that has been filtered out a file that has been filtered out
--break-per-input --break-on-existing, --break-on-reject, --break-per-input Alters --max-downloads, --break-on-existing,
--max-downloads, and autonumber resets per --break-on-reject, and autonumber to reset
input URL per input URL
--no-break-per-input --break-on-existing and similar options --no-break-per-input --break-on-existing and similar options
terminates the entire download queue terminates the entire download queue
--skip-playlist-after-errors N Number of allowed failures until the rest of --skip-playlist-after-errors N Number of allowed failures until the rest of
@ -1046,10 +1046,10 @@ ## SponsorBlock Options:
for, separated by commas. Available for, separated by commas. Available
categories are sponsor, intro, outro, categories are sponsor, intro, outro,
selfpromo, preview, filler, interaction, selfpromo, preview, filler, interaction,
music_offtopic, poi_highlight, chapter, all and music_offtopic, poi_highlight, chapter, all
default (=all). You can prefix the category and default (=all). You can prefix the
with a "-" to exclude it. See [1] for category with a "-" to exclude it. See [1]
description of the categories. E.g. for description of the categories. E.g.
--sponsorblock-mark all,-preview --sponsorblock-mark all,-preview
[1] https://wiki.sponsor.ajay.app/w/Segment_Categories [1] https://wiki.sponsor.ajay.app/w/Segment_Categories
--sponsorblock-remove CATS SponsorBlock categories to be removed from --sponsorblock-remove CATS SponsorBlock categories to be removed from
@ -1058,7 +1058,7 @@ ## SponsorBlock Options:
remove takes precedence. The syntax and remove takes precedence. The syntax and
available categories are the same as for available categories are the same as for
--sponsorblock-mark except that "default" --sponsorblock-mark except that "default"
refers to "all,-filler" and poi_highlight and refers to "all,-filler" and poi_highlight,
chapter are not available chapter are not available
--sponsorblock-chapter-title TEMPLATE --sponsorblock-chapter-title TEMPLATE
An output template for the title of the An output template for the title of the

View file

@ -3123,7 +3123,7 @@ def existing_video_file(*filepaths):
fd, success = None, True fd, success = None, True
if info_dict.get('protocol') or info_dict.get('url'): if info_dict.get('protocol') or info_dict.get('url'):
fd = get_suitable_downloader(info_dict, self.params, to_stdout=temp_filename == '-') fd = get_suitable_downloader(info_dict, self.params, to_stdout=temp_filename == '-')
if fd is not FFmpegFD and ( if fd is not FFmpegFD and 'no-direct-merge' not in self.params['compat_opts'] and (
info_dict.get('section_start') or info_dict.get('section_end')): info_dict.get('section_start') or info_dict.get('section_end')):
msg = ('This format cannot be partially downloaded' if FFmpegFD.available() msg = ('This format cannot be partially downloaded' if FFmpegFD.available()
else 'You have requested downloading the video partially, but ffmpeg is not installed') else 'You have requested downloading the video partially, but ffmpeg is not installed')

View file

@ -91,12 +91,11 @@ def get_urls(urls, batchfile, verbose):
def print_extractor_information(opts, urls): def print_extractor_information(opts, urls):
# Importing GenericIE is currently slow since it imports other extractors
# TODO: Move this back to module level after generalization of embed detection
from .extractor.generic import GenericIE
out = '' out = ''
if opts.list_extractors: if opts.list_extractors:
# Importing GenericIE is currently slow since it imports YoutubeIE
from .extractor.generic import GenericIE
urls = dict.fromkeys(urls, False) urls = dict.fromkeys(urls, False)
for ie in list_extractor_classes(opts.age_limit): for ie in list_extractor_classes(opts.age_limit):
out += ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie.working() else '') + '\n' out += ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie.working() else '') + '\n'

View file

@ -20,6 +20,7 @@
RetryManager, RetryManager,
classproperty, classproperty,
decodeArgument, decodeArgument,
deprecation_warning,
encodeFilename, encodeFilename,
format_bytes, format_bytes,
join_nonempty, join_nonempty,
@ -180,7 +181,9 @@ def best_block_size(elapsed_time, bytes):
@staticmethod @staticmethod
def parse_bytes(bytestr): def parse_bytes(bytestr):
"""Parse a string indicating a byte quantity into an integer.""" """Parse a string indicating a byte quantity into an integer."""
parse_bytes(bytestr) deprecation_warning('yt_dlp.FileDownloader.parse_bytes is deprecated and '
'may be removed in the future. Use yt_dlp.utils.parse_bytes instead')
return parse_bytes(bytestr)
def slow_down(self, start_time, now, byte_counter): def slow_down(self, start_time, now, byte_counter):
"""Sleep if the download speed is over the rate limit.""" """Sleep if the download speed is over the rate limit."""

View file

@ -71,6 +71,7 @@
str_to_int, str_to_int,
strip_or_none, strip_or_none,
traverse_obj, traverse_obj,
truncate_string,
try_call, try_call,
try_get, try_get,
unescapeHTML, unescapeHTML,
@ -674,7 +675,8 @@ def extract(self, url):
for _ in range(2): for _ in range(2):
try: try:
self.initialize() self.initialize()
self.write_debug('Extracting URL: %s' % url) self.to_screen('Extracting URL: %s' % (
url if self.get_param('verbose') else truncate_string(url, 100, 20)))
ie_result = self._real_extract(url) ie_result = self._real_extract(url)
if ie_result is None: if ie_result is None:
return None return None
@ -1906,6 +1908,14 @@ def _extract_m3u8_formats_and_subtitles(
errnote=None, fatal=True, live=False, data=None, headers={}, errnote=None, fatal=True, live=False, data=None, headers={},
query={}): query={}):
if not m3u8_url:
if errnote is not False:
errnote = errnote or 'Failed to obtain m3u8 URL'
if fatal:
raise ExtractorError(errnote, video_id=video_id)
self.report_warning(f'{errnote}{bug_reports_message()}')
return [], {}
res = self._download_webpage_handle( res = self._download_webpage_handle(
m3u8_url, video_id, m3u8_url, video_id,
note='Downloading m3u8 information' if note is None else note, note='Downloading m3u8 information' if note is None else note,

View file

@ -535,10 +535,10 @@ def _alias_callback(option, opt_str, value, parser, opts, nargs):
'-I', '--playlist-items', '-I', '--playlist-items',
dest='playlist_items', metavar='ITEM_SPEC', default=None, dest='playlist_items', metavar='ITEM_SPEC', default=None,
help=( help=(
'Comma separated playlist_index of the videos to download. ' 'Comma separated playlist_index of the items to download. '
'You can specify a range using "[START]:[STOP][:STEP]". For backward compatibility, START-STOP is also supported. ' 'You can specify a range using "[START]:[STOP][:STEP]". For backward compatibility, START-STOP is also supported. '
'Use negative indices to count from the right and negative STEP to download in reverse order. ' 'Use negative indices to count from the right and negative STEP to download in reverse order. '
'E.g. "-I 1:3,7,-5::2" used on a playlist of size 15 will download the videos at index 1,2,3,7,11,13,15')) 'E.g. "-I 1:3,7,-5::2" used on a playlist of size 15 will download the items at index 1,2,3,7,11,13,15'))
selection.add_option( selection.add_option(
'--match-title', '--match-title',
dest='matchtitle', metavar='REGEX', dest='matchtitle', metavar='REGEX',
@ -554,7 +554,7 @@ def _alias_callback(option, opt_str, value, parser, opts, nargs):
selection.add_option( selection.add_option(
'--max-filesize', '--max-filesize',
metavar='SIZE', dest='max_filesize', default=None, metavar='SIZE', dest='max_filesize', default=None,
help='Abort download if filesize if larger than SIZE, e.g. 50k or 44.6M') help='Abort download if filesize is larger than SIZE, e.g. 50k or 44.6M')
selection.add_option( selection.add_option(
'--date', '--date',
metavar='DATE', dest='date', default=None, metavar='DATE', dest='date', default=None,
@ -635,7 +635,7 @@ def _alias_callback(option, opt_str, value, parser, opts, nargs):
selection.add_option( selection.add_option(
'--break-per-input', '--break-per-input',
action='store_true', dest='break_per_url', default=False, action='store_true', dest='break_per_url', default=False,
help='--break-on-existing, --break-on-reject, --max-downloads, and autonumber resets per input URL') help='Alters --max-downloads, --break-on-existing, --break-on-reject, and autonumber to reset per input URL')
selection.add_option( selection.add_option(
'--no-break-per-input', '--no-break-per-input',
action='store_false', dest='break_per_url', action='store_false', dest='break_per_url',

View file

@ -3872,6 +3872,9 @@ def __eq__(self, other):
return (isinstance(other, download_range_func) return (isinstance(other, download_range_func)
and self.chapters == other.chapters and self.ranges == other.ranges) and self.chapters == other.chapters and self.ranges == other.ranges)
def __repr__(self):
return f'{type(self).__name__}({self.chapters}, {self.ranges})'
def parse_dfxp_time_expr(time_expr): def parse_dfxp_time_expr(time_expr):
if not time_expr: if not time_expr:
@ -5976,7 +5979,7 @@ def truncate_string(s, left, right=0):
assert left > 3 and right >= 0 assert left > 3 and right >= 0
if s is None or len(s) <= left + right: if s is None or len(s) <= left + right:
return s return s
return f'{s[:left-3]}...{s[-right:]}' return f'{s[:left-3]}...{s[-right:] if right else ""}'
def orderedSet_from_options(options, alias_dict, *, use_regex=False, start=None): def orderedSet_from_options(options, alias_dict, *, use_regex=False, start=None):