mirror of
https://github.com/python/cpython
synced 2024-09-04 15:56:13 +00:00
![Thomas Wouters](/assets/img/avatar_default.png)
svn+ssh://pythondev@svn.python.org/python/trunk ........ r57771 | thomas.wouters | 2007-08-30 23:54:39 +0200 (Thu, 30 Aug 2007) | 5 lines Don't lie in __all__ attributes when SSL is not available: only add the SSL classes when they are actually created. ........ r57620 | walter.doerwald | 2007-08-28 18:38:26 +0200 (Tue, 28 Aug 2007) | 5 lines Fix title endtag in HTMLCalender.formatyearpage(). Fix documentation for HTMLCalender.formatyearpage() (there's no themonth parameter). This fixes issue1046. ........ r57622 | georg.brandl | 2007-08-28 20:54:44 +0200 (Tue, 28 Aug 2007) | 2 lines Add a crasher for the thread-unsafety of file objects. ........ r57626 | skip.montanaro | 2007-08-29 01:22:52 +0200 (Wed, 29 Aug 2007) | 1 line fixes 813986 ........ r57628 | walter.doerwald | 2007-08-29 01:35:33 +0200 (Wed, 29 Aug 2007) | 2 lines Fix test output. ........ r57631 | skip.montanaro | 2007-08-29 03:24:11 +0200 (Wed, 29 Aug 2007) | 2 lines Install pygettext (once the scriptsinstall target is working again). ........ r57633 | skip.montanaro | 2007-08-29 03:33:45 +0200 (Wed, 29 Aug 2007) | 2 lines Recent items. ........ r57650 | neal.norwitz | 2007-08-29 08:15:33 +0200 (Wed, 29 Aug 2007) | 1 line Add Bill as a developer ........ r57651 | facundo.batista | 2007-08-29 12:28:28 +0200 (Wed, 29 Aug 2007) | 5 lines Ignore test failures caused by 'resource temporarily unavailable' exceptions raised during FailingServerTestCase tests. [GSoC - Alan McIntyre] ........ r57680 | bill.janssen | 2007-08-30 00:35:05 +0200 (Thu, 30 Aug 2007) | 17 lines This contains a number of things: 1) Improve the documentation of the SSL module, with a fuller explanation of certificate usage, another reference, proper formatting of this and that. 2) Fix Windows bug in ssl.py, and general bug in sslsocket.close(). Remove some unused code from ssl.py. Allow accept() to be called on sslsocket sockets. 3) Use try-except-else in import of ssl in socket.py. Deprecate use of socket.ssl(). 4) Remove use of socket.ssl() in every library module, except for test_socket_ssl.py and test_ssl.py. ........ r57714 | georg.brandl | 2007-08-30 12:09:42 +0200 (Thu, 30 Aug 2007) | 2 lines Stronger urge to convert filenames to str before using them as argument to ZipFile.write(). ........ r57716 | georg.brandl | 2007-08-30 12:38:56 +0200 (Thu, 30 Aug 2007) | 2 lines Patch #1680959: add test suite for pipes module. ........ r57717 | georg.brandl | 2007-08-30 14:32:23 +0200 (Thu, 30 Aug 2007) | 3 lines * Skip test_pipes on non-POSIX. * Don't raise TestSkipped within a test function. ........ r57723 | mark.summerfield | 2007-08-30 17:03:03 +0200 (Thu, 30 Aug 2007) | 3 lines Added more cross-references. ........ r57726 | walter.doerwald | 2007-08-30 17:30:09 +0200 (Thu, 30 Aug 2007) | 2 lines Rewrap line. ........ r57727 | walter.doerwald | 2007-08-30 17:34:55 +0200 (Thu, 30 Aug 2007) | 2 lines Set startinpos before calling the error handler. ........ r57730 | bill.janssen | 2007-08-30 19:07:28 +0200 (Thu, 30 Aug 2007) | 3 lines Added docstrings to methods and functions. ........ r57743 | bill.janssen | 2007-08-30 20:08:06 +0200 (Thu, 30 Aug 2007) | 1 line added note on new ssl module and deprecation of socket.ssl ........ r57747 | martin.v.loewis | 2007-08-30 20:14:01 +0200 (Thu, 30 Aug 2007) | 1 line Fix popen usage. ........ r57748 | martin.v.loewis | 2007-08-30 20:15:22 +0200 (Thu, 30 Aug 2007) | 1 line Fix typo. ........ r57750 | martin.v.loewis | 2007-08-30 20:25:47 +0200 (Thu, 30 Aug 2007) | 1 line Bug #1746880: Correctly install DLLs into system32 folder on Win64. ........ r57760 | martin.v.loewis | 2007-08-30 21:04:09 +0200 (Thu, 30 Aug 2007) | 1 line Bug #1709599: Run test_1565150 only if the file system is NTFS. ........ r57762 | martin.v.loewis | 2007-08-30 22:10:57 +0200 (Thu, 30 Aug 2007) | 2 lines Bump autoconf minimum version to 2.61. ........ r57764 | lars.gustaebel | 2007-08-30 22:24:31 +0200 (Thu, 30 Aug 2007) | 2 lines Warn about possible risks when extracting untrusted archives. ........ r57769 | thomas.wouters | 2007-08-30 23:01:17 +0200 (Thu, 30 Aug 2007) | 7 lines Somewhat-preliminary slice-object and extended slicing support for ctypes. The exact behaviour of omitted and negative indices for the Pointer type may need a closer look (especially as it's subtly different from simple slices) but there's time yet before 2.6, and not enough before 3.0a1 :-) ........
269 lines
9.1 KiB
Python
269 lines
9.1 KiB
Python
"""Conversion pipeline templates.
|
|
|
|
The problem:
|
|
------------
|
|
|
|
Suppose you have some data that you want to convert to another format,
|
|
such as from GIF image format to PPM image format. Maybe the
|
|
conversion involves several steps (e.g. piping it through compress or
|
|
uuencode). Some of the conversion steps may require that their input
|
|
is a disk file, others may be able to read standard input; similar for
|
|
their output. The input to the entire conversion may also be read
|
|
from a disk file or from an open file, and similar for its output.
|
|
|
|
The module lets you construct a pipeline template by sticking one or
|
|
more conversion steps together. It will take care of creating and
|
|
removing temporary files if they are necessary to hold intermediate
|
|
data. You can then use the template to do conversions from many
|
|
different sources to many different destinations. The temporary
|
|
file names used are different each time the template is used.
|
|
|
|
The templates are objects so you can create templates for many
|
|
different conversion steps and store them in a dictionary, for
|
|
instance.
|
|
|
|
|
|
Directions:
|
|
-----------
|
|
|
|
To create a template:
|
|
t = Template()
|
|
|
|
To add a conversion step to a template:
|
|
t.append(command, kind)
|
|
where kind is a string of two characters: the first is '-' if the
|
|
command reads its standard input or 'f' if it requires a file; the
|
|
second likewise for the output. The command must be valid /bin/sh
|
|
syntax. If input or output files are required, they are passed as
|
|
$IN and $OUT; otherwise, it must be possible to use the command in
|
|
a pipeline.
|
|
|
|
To add a conversion step at the beginning:
|
|
t.prepend(command, kind)
|
|
|
|
To convert a file to another file using a template:
|
|
sts = t.copy(infile, outfile)
|
|
If infile or outfile are the empty string, standard input is read or
|
|
standard output is written, respectively. The return value is the
|
|
exit status of the conversion pipeline.
|
|
|
|
To open a file for reading or writing through a conversion pipeline:
|
|
fp = t.open(file, mode)
|
|
where mode is 'r' to read the file, or 'w' to write it -- just like
|
|
for the built-in function open() or for os.popen().
|
|
|
|
To create a new template object initialized to a given one:
|
|
t2 = t.clone()
|
|
|
|
For an example, see the function test() at the end of the file.
|
|
""" # '
|
|
|
|
|
|
import re
|
|
import os
|
|
import tempfile
|
|
import string
|
|
|
|
__all__ = ["Template"]
|
|
|
|
# Conversion step kinds
|
|
|
|
FILEIN_FILEOUT = 'ff' # Must read & write real files
|
|
STDIN_FILEOUT = '-f' # Must write a real file
|
|
FILEIN_STDOUT = 'f-' # Must read a real file
|
|
STDIN_STDOUT = '--' # Normal pipeline element
|
|
SOURCE = '.-' # Must be first, writes stdout
|
|
SINK = '-.' # Must be last, reads stdin
|
|
|
|
stepkinds = [FILEIN_FILEOUT, STDIN_FILEOUT, FILEIN_STDOUT, STDIN_STDOUT, \
|
|
SOURCE, SINK]
|
|
|
|
|
|
class Template:
|
|
"""Class representing a pipeline template."""
|
|
|
|
def __init__(self):
|
|
"""Template() returns a fresh pipeline template."""
|
|
self.debugging = 0
|
|
self.reset()
|
|
|
|
def __repr__(self):
|
|
"""t.__repr__() implements repr(t)."""
|
|
return '<Template instance, steps=%r>' % (self.steps,)
|
|
|
|
def reset(self):
|
|
"""t.reset() restores a pipeline template to its initial state."""
|
|
self.steps = []
|
|
|
|
def clone(self):
|
|
"""t.clone() returns a new pipeline template with identical
|
|
initial state as the current one."""
|
|
t = Template()
|
|
t.steps = self.steps[:]
|
|
t.debugging = self.debugging
|
|
return t
|
|
|
|
def debug(self, flag):
|
|
"""t.debug(flag) turns debugging on or off."""
|
|
self.debugging = flag
|
|
|
|
def append(self, cmd, kind):
|
|
"""t.append(cmd, kind) adds a new step at the end."""
|
|
if type(cmd) is not type(''):
|
|
raise TypeError('Template.append: cmd must be a string')
|
|
if kind not in stepkinds:
|
|
raise ValueError('Template.append: bad kind %r' % (kind,))
|
|
if kind == SOURCE:
|
|
raise ValueError('Template.append: SOURCE can only be prepended')
|
|
if self.steps and self.steps[-1][1] == SINK:
|
|
raise ValueError('Template.append: already ends with SINK')
|
|
if kind[0] == 'f' and not re.search(r'\$IN\b', cmd):
|
|
raise ValueError('Template.append: missing $IN in cmd')
|
|
if kind[1] == 'f' and not re.search(r'\$OUT\b', cmd):
|
|
raise ValueError('Template.append: missing $OUT in cmd')
|
|
self.steps.append((cmd, kind))
|
|
|
|
def prepend(self, cmd, kind):
|
|
"""t.prepend(cmd, kind) adds a new step at the front."""
|
|
if type(cmd) is not type(''):
|
|
raise TypeError('Template.prepend: cmd must be a string')
|
|
if kind not in stepkinds:
|
|
raise ValueError('Template.prepend: bad kind %r' % (kind,))
|
|
if kind == SINK:
|
|
raise ValueError('Template.prepend: SINK can only be appended')
|
|
if self.steps and self.steps[0][1] == SOURCE:
|
|
raise ValueError('Template.prepend: already begins with SOURCE')
|
|
if kind[0] == 'f' and not re.search(r'\$IN\b', cmd):
|
|
raise ValueError('Template.prepend: missing $IN in cmd')
|
|
if kind[1] == 'f' and not re.search(r'\$OUT\b', cmd):
|
|
raise ValueError('Template.prepend: missing $OUT in cmd')
|
|
self.steps.insert(0, (cmd, kind))
|
|
|
|
def open(self, file, rw):
|
|
"""t.open(file, rw) returns a pipe or file object open for
|
|
reading or writing; the file is the other end of the pipeline."""
|
|
if rw == 'r':
|
|
return self.open_r(file)
|
|
if rw == 'w':
|
|
return self.open_w(file)
|
|
raise ValueError('Template.open: rw must be \'r\' or \'w\', not %r'
|
|
% (rw,))
|
|
|
|
def open_r(self, file):
|
|
"""t.open_r(file) and t.open_w(file) implement
|
|
t.open(file, 'r') and t.open(file, 'w') respectively."""
|
|
if not self.steps:
|
|
return open(file, 'r')
|
|
if self.steps[-1][1] == SINK:
|
|
raise ValueError('Template.open_r: pipeline ends width SINK')
|
|
cmd = self.makepipeline(file, '')
|
|
return os.popen(cmd, 'r')
|
|
|
|
def open_w(self, file):
|
|
if not self.steps:
|
|
return open(file, 'w')
|
|
if self.steps[0][1] == SOURCE:
|
|
raise ValueError('Template.open_w: pipeline begins with SOURCE')
|
|
cmd = self.makepipeline('', file)
|
|
return os.popen(cmd, 'w')
|
|
|
|
def copy(self, infile, outfile):
|
|
return os.system(self.makepipeline(infile, outfile))
|
|
|
|
def makepipeline(self, infile, outfile):
|
|
cmd = makepipeline(infile, self.steps, outfile)
|
|
if self.debugging:
|
|
print(cmd)
|
|
cmd = 'set -x; ' + cmd
|
|
return cmd
|
|
|
|
|
|
def makepipeline(infile, steps, outfile):
|
|
# Build a list with for each command:
|
|
# [input filename or '', command string, kind, output filename or '']
|
|
|
|
list = []
|
|
for cmd, kind in steps:
|
|
list.append(['', cmd, kind, ''])
|
|
#
|
|
# Make sure there is at least one step
|
|
#
|
|
if not list:
|
|
list.append(['', 'cat', '--', ''])
|
|
#
|
|
# Take care of the input and output ends
|
|
#
|
|
[cmd, kind] = list[0][1:3]
|
|
if kind[0] == 'f' and not infile:
|
|
list.insert(0, ['', 'cat', '--', ''])
|
|
list[0][0] = infile
|
|
#
|
|
[cmd, kind] = list[-1][1:3]
|
|
if kind[1] == 'f' and not outfile:
|
|
list.append(['', 'cat', '--', ''])
|
|
list[-1][-1] = outfile
|
|
#
|
|
# Invent temporary files to connect stages that need files
|
|
#
|
|
garbage = []
|
|
for i in range(1, len(list)):
|
|
lkind = list[i-1][2]
|
|
rkind = list[i][2]
|
|
if lkind[1] == 'f' or rkind[0] == 'f':
|
|
(fd, temp) = tempfile.mkstemp()
|
|
os.close(fd)
|
|
garbage.append(temp)
|
|
list[i-1][-1] = list[i][0] = temp
|
|
#
|
|
for item in list:
|
|
[inf, cmd, kind, outf] = item
|
|
if kind[1] == 'f':
|
|
cmd = 'OUT=' + quote(outf) + '; ' + cmd
|
|
if kind[0] == 'f':
|
|
cmd = 'IN=' + quote(inf) + '; ' + cmd
|
|
if kind[0] == '-' and inf:
|
|
cmd = cmd + ' <' + quote(inf)
|
|
if kind[1] == '-' and outf:
|
|
cmd = cmd + ' >' + quote(outf)
|
|
item[1] = cmd
|
|
#
|
|
cmdlist = list[0][1]
|
|
for item in list[1:]:
|
|
[cmd, kind] = item[1:3]
|
|
if item[0] == '':
|
|
if 'f' in kind:
|
|
cmd = '{ ' + cmd + '; }'
|
|
cmdlist = cmdlist + ' |\n' + cmd
|
|
else:
|
|
cmdlist = cmdlist + '\n' + cmd
|
|
#
|
|
if garbage:
|
|
rmcmd = 'rm -f'
|
|
for file in garbage:
|
|
rmcmd = rmcmd + ' ' + quote(file)
|
|
trapcmd = 'trap ' + quote(rmcmd + '; exit') + ' 1 2 3 13 14 15'
|
|
cmdlist = trapcmd + '\n' + cmdlist + '\n' + rmcmd
|
|
#
|
|
return cmdlist
|
|
|
|
|
|
# Reliably quote a string as a single argument for /bin/sh
|
|
|
|
_safechars = string.ascii_letters + string.digits + '!@%_-+=:,./' # Safe unquoted
|
|
_funnychars = '"`$\\' # Unsafe inside "double quotes"
|
|
|
|
def quote(file):
|
|
for c in file:
|
|
if c not in _safechars:
|
|
break
|
|
else:
|
|
return file
|
|
if '\'' not in file:
|
|
return '\'' + file + '\''
|
|
res = ''
|
|
for c in file:
|
|
if c in _funnychars:
|
|
c = '\\' + c
|
|
res = res + c
|
|
return '"' + res + '"'
|