From 4a4be2dca8aa4d326f62ba71d01faa95da282bb4 Mon Sep 17 00:00:00 2001 From: delirious-lettuce Date: Fri, 19 May 2017 14:37:57 -0600 Subject: [PATCH 1/2] Resolved conflict --- Doc/c-api/exceptions.rst | 10 +- Doc/howto/logging-cookbook.rst | 2 +- Doc/library/cmd.rst | 165 ++++++ Doc/library/email.compat32-message.rst | 758 +++++++++++++++++++++++++ Doc/library/email.contentmanager.rst | 207 +++++++ Doc/library/email.errors.rst | 10 + Doc/library/email.generator.rst | 101 ++++ Doc/library/email.headerregistry.rst | 455 +++++++++++++++ Doc/library/email.message.rst | 212 +++++++ Doc/library/email.mime.rst | 2 +- Doc/library/email.parser.rst | 6 + Doc/library/email.policy.rst | 651 +++++++++++++++++++++ Doc/library/exceptions.rst | 6 + Doc/library/sunau.rst | 2 +- Doc/library/turtle.rst | 50 ++ Doc/library/xml.dom.pulldom.rst | 10 + Doc/reference/expressions.rst | 118 ++++ Lib/email/architecture.rst | 216 +++++++ 18 files changed, 2975 insertions(+), 6 deletions(-) create mode 100644 Doc/library/email.compat32-message.rst create mode 100644 Doc/library/email.contentmanager.rst create mode 100644 Doc/library/email.headerregistry.rst create mode 100644 Doc/library/email.policy.rst create mode 100644 Lib/email/architecture.rst diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index d9e333311b595b..5522ca2e23b8d6 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -290,7 +290,7 @@ is a separate error indicator for each thread. :c:data:`PyExc_Warning` is a subclass of :c:data:`PyExc_Exception`; the default warning category is :c:data:`PyExc_RuntimeWarning`. The standard Python warning categories are available as global variables whose names are - enumerated at :ref:`standarwarningcategories`. + enumerated at :ref:`standardwarningcategories`. For information about warning control, see the documentation for the :mod:`warnings` module and the :option:`-W` option in the command line @@ -658,6 +658,7 @@ Notes: Only defined on Windows; protect code that uses this by testing that the preprocessor macro ``MS_WINDOWS`` is defined. +<<<<<<< HEAD (4) .. versionadded:: 2.5 @@ -666,6 +667,9 @@ Notes: preprocessor macro ``__VMS`` is defined. .. _standarwarningcategories: +======= +.. _standardwarningcategories: +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) Standard Warning Categories =========================== @@ -678,7 +682,7 @@ the variables: .. index:: single: PyExc_Warning single: PyExc_BytesWarning - single: PyExc_DepricationWarning + single: PyExc_DeprecationWarning single: PyExc_FutureWarning single: PyExc_ImportWarning single: PyExc_PendingDeprecationWarning @@ -700,7 +704,7 @@ the variables: +------------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_ImportWarning` | :exc:`ImportWarning` | | +------------------------------------------+---------------------------------+----------+ -| :c:data:`PyExc_PendingDepricationWarning`| :exc:`PendingDeprecationWarning`| | +| :c:data:`PyExc_PendingDeprecationWarning`| :exc:`PendingDeprecationWarning`| | +------------------------------------------+---------------------------------+----------+ | :c:data:`PyExc_RuntimeWarning` | :exc:`RuntimeWarning` | | +------------------------------------------+---------------------------------+----------+ diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index 0818fda682eb10..b9b5120f053d8f 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -814,7 +814,7 @@ Implementing structured logging ------------------------------- Although most logging messages are intended for reading by humans, and thus not -readily machine-parseable, there might be cirumstances where you want to output +readily machine-parseable, there might be circumstances where you want to output messages in a structured format which *is* capable of being parsed by a program (without needing complex regular expressions to parse the log message). This is straightforward to achieve using the logging package. There are a number of diff --git a/Doc/library/cmd.rst b/Doc/library/cmd.rst index 05341d7cb04f7c..07f02020ceb1da 100644 --- a/Doc/library/cmd.rst +++ b/Doc/library/cmd.rst @@ -215,3 +215,168 @@ Instances of :class:`Cmd` subclasses have some public instance variables: :mod:`readline`, on systems that support it, the interpreter will automatically support :program:`Emacs`\ -like line editing and command-history keystrokes.) +<<<<<<< HEAD +======= + +.. _cmd-example: + +Cmd Example +----------- + +.. sectionauthor:: Raymond Hettinger + +The :mod:`cmd` module is mainly useful for building custom shells that let a +user work with a program interactively. + +This section presents a simple example of how to build a shell around a few of +the commands in the :mod:`turtle` module. + +Basic turtle commands such as :meth:`~turtle.forward` are added to a +:class:`Cmd` subclass with method named :meth:`do_forward`. The argument is +converted to a number and dispatched to the turtle module. The docstring is +used in the help utility provided by the shell. + +The example also includes a basic record and playback facility implemented with +the :meth:`~Cmd.precmd` method which is responsible for converting the input to +lowercase and writing the commands to a file. The :meth:`do_playback` method +reads the file and adds the recorded commands to the :attr:`cmdqueue` for +immediate playback:: + + import cmd, sys + from turtle import * + + class TurtleShell(cmd.Cmd): + intro = 'Welcome to the turtle shell. Type help or ? to list commands.\n' + prompt = '(turtle) ' + file = None + + # ----- basic turtle commands ----- + def do_forward(self, arg): + 'Move the turtle forward by the specified distance: FORWARD 10' + forward(*parse(arg)) + def do_right(self, arg): + 'Turn turtle right by given number of degrees: RIGHT 20' + right(*parse(arg)) + def do_left(self, arg): + 'Turn turtle left by given number of degrees: LEFT 90' + left(*parse(arg)) + def do_goto(self, arg): + 'Move turtle to an absolute position with changing orientation. GOTO 100 200' + goto(*parse(arg)) + def do_home(self, arg): + 'Return turtle to the home position: HOME' + home() + def do_circle(self, arg): + 'Draw circle with given radius an options extent and steps: CIRCLE 50' + circle(*parse(arg)) + def do_position(self, arg): + 'Print the current turtle position: POSITION' + print('Current position is %d %d\n' % position()) + def do_heading(self, arg): + 'Print the current turtle heading in degrees: HEADING' + print('Current heading is %d\n' % (heading(),)) + def do_color(self, arg): + 'Set the color: COLOR BLUE' + color(arg.lower()) + def do_undo(self, arg): + 'Undo (repeatedly) the last turtle action(s): UNDO' + def do_reset(self, arg): + 'Clear the screen and return turtle to center: RESET' + reset() + def do_bye(self, arg): + 'Stop recording, close the turtle window, and exit: BYE' + print('Thank you for using Turtle') + self.close() + bye() + return True + + # ----- record and playback ----- + def do_record(self, arg): + 'Save future commands to filename: RECORD rose.cmd' + self.file = open(arg, 'w') + def do_playback(self, arg): + 'Playback commands from a file: PLAYBACK rose.cmd' + self.close() + with open(arg) as f: + self.cmdqueue.extend(f.read().splitlines()) + def precmd(self, line): + line = line.lower() + if self.file and 'playback' not in line: + print(line, file=self.file) + return line + def close(self): + if self.file: + self.file.close() + self.file = None + + def parse(arg): + 'Convert a series of zero or more numbers to an argument tuple' + return tuple(map(int, arg.split())) + + if __name__ == '__main__': + TurtleShell().cmdloop() + + +Here is a sample session with the turtle shell showing the help functions, using +blank lines to repeat commands, and the simple record and playback facility: + +.. code-block:: none + + Welcome to the turtle shell. Type help or ? to list commands. + + (turtle) ? + + Documented commands (type help ): + ======================================== + bye color goto home playback record right + circle forward heading left position reset undo + + (turtle) help forward + Move the turtle forward by the specified distance: FORWARD 10 + (turtle) record spiral.cmd + (turtle) position + Current position is 0 0 + + (turtle) heading + Current heading is 0 + + (turtle) reset + (turtle) circle 20 + (turtle) right 30 + (turtle) circle 40 + (turtle) right 30 + (turtle) circle 60 + (turtle) right 30 + (turtle) circle 80 + (turtle) right 30 + (turtle) circle 100 + (turtle) right 30 + (turtle) circle 120 + (turtle) right 30 + (turtle) circle 120 + (turtle) heading + Current heading is 180 + + (turtle) forward 100 + (turtle) + (turtle) right 90 + (turtle) forward 100 + (turtle) + (turtle) right 90 + (turtle) forward 400 + (turtle) right 90 + (turtle) forward 500 + (turtle) right 90 + (turtle) forward 400 + (turtle) right 90 + (turtle) forward 300 + (turtle) playback spiral.cmd + Current position is 0 0 + + Current heading is 0 + + Current heading is 180 + + (turtle) bye + Thank you for using Turtle +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) diff --git a/Doc/library/email.compat32-message.rst b/Doc/library/email.compat32-message.rst new file mode 100644 index 00000000000000..6f8e489a7328e6 --- /dev/null +++ b/Doc/library/email.compat32-message.rst @@ -0,0 +1,758 @@ +.. _compat32_message: + +:mod:`email.message.Message`: Representing an email message using the :data:`~email.policy.compat32` API +-------------------------------------------------------------------------------------------------------- + +.. module:: email.message + :synopsis: The base class representing email messages in a fashion + backward compatible with python3.2 + + +The :class:`Message` class is very similar to the +:class:`~email.message.EmailMessage` class, without the methods added by that +class, and with the default behavior of certain other methods being slightly +different. We also document here some methods that, while supported by the +:class:`~email.message.EmailMessage` class, are not recommended unless you are +dealing with legacy code. + +The philosophy and structure of the two classes is otherwise the same. + +This document describes the behavior under the default (for :class:`Message`) +policy :attr:`~email.policy.Compat32`. If you are going to use another policy, +you should be using the :class:`~email.message.EmailMessage` class instead. + +An email message consists of *headers* and a *payload*. Headers must be +:rfc:`5233` style names and values, where the field name and value are +separated by a colon. The colon is not part of either the field name or the +field value. The payload may be a simple text message, or a binary object, or +a structured sequence of sub-messages each with their own set of headers and +their own payload. The latter type of payload is indicated by the message +having a MIME type such as :mimetype:`multipart/\*` or +:mimetype:`message/rfc822`. + +The conceptual model provided by a :class:`Message` object is that of an +ordered dictionary of headers with additional methods for accessing both +specialized information from the headers, for accessing the payload, for +generating a serialized version of the message, and for recursively walking +over the object tree. Note that duplicate headers are supported but special +methods must be used to access them. + +The :class:`Message` pseudo-dictionary is indexed by the header names, which +must be ASCII values. The values of the dictionary are strings that are +supposed to contain only ASCII characters; there is some special handling for +non-ASCII input, but it doesn't always produce the correct results. Headers +are stored and returned in case-preserving form, but field names are matched +case-insensitively. There may also be a single envelope header, also known as +the *Unix-From* header or the ``From_`` header. The *payload* is either a +string or bytes, in the case of simple message objects, or a list of +:class:`Message` objects, for MIME container documents (e.g. +:mimetype:`multipart/\*` and :mimetype:`message/rfc822`). + +Here are the methods of the :class:`Message` class: + + +.. class:: Message(policy=compat32) + + If *policy* is specified (it must be an instance of a :mod:`~email.policy` + class) use the rules it specifies to update and serialize the representation + of the message. If *policy* is not set, use the :class:`compat32 + ` policy, which maintains backward compatibility with + the Python 3.2 version of the email package. For more information see the + :mod:`~email.policy` documentation. + + .. versionchanged:: 3.3 The *policy* keyword argument was added. + + + .. method:: as_string(unixfrom=False, maxheaderlen=0, policy=None) + + Return the entire message flattened as a string. When optional *unixfrom* + is true, the envelope header is included in the returned string. + *unixfrom* defaults to ``False``. For backward compatibility reasons, + *maxheaderlen* defaults to ``0``, so if you want a different value you + must override it explicitly (the value specified for *max_line_length* in + the policy will be ignored by this method). The *policy* argument may be + used to override the default policy obtained from the message instance. + This can be used to control some of the formatting produced by the + method, since the specified *policy* will be passed to the ``Generator``. + + Flattening the message may trigger changes to the :class:`Message` if + defaults need to be filled in to complete the transformation to a string + (for example, MIME boundaries may be generated or modified). + + Note that this method is provided as a convenience and may not always + format the message the way you want. For example, by default it does + not do the mangling of lines that begin with ``From`` that is + required by the unix mbox format. For more flexibility, instantiate a + :class:`~email.generator.Generator` instance and use its + :meth:`~email.generator.Generator.flatten` method directly. For example:: + + from io import StringIO + from email.generator import Generator + fp = StringIO() + g = Generator(fp, mangle_from_=True, maxheaderlen=60) + g.flatten(msg) + text = fp.getvalue() + + If the message object contains binary data that is not encoded according + to RFC standards, the non-compliant data will be replaced by unicode + "unknown character" code points. (See also :meth:`.as_bytes` and + :class:`~email.generator.BytesGenerator`.) + + .. versionchanged:: 3.4 the *policy* keyword argument was added. + + + .. method:: __str__() + + Equivalent to :meth:`.as_string()`. Allows ``str(msg)`` to produce a + string containing the formatted message. + + + .. method:: as_bytes(unixfrom=False, policy=None) + + Return the entire message flattened as a bytes object. When optional + *unixfrom* is true, the envelope header is included in the returned + string. *unixfrom* defaults to ``False``. The *policy* argument may be + used to override the default policy obtained from the message instance. + This can be used to control some of the formatting produced by the + method, since the specified *policy* will be passed to the + ``BytesGenerator``. + + Flattening the message may trigger changes to the :class:`Message` if + defaults need to be filled in to complete the transformation to a string + (for example, MIME boundaries may be generated or modified). + + Note that this method is provided as a convenience and may not always + format the message the way you want. For example, by default it does + not do the mangling of lines that begin with ``From`` that is + required by the unix mbox format. For more flexibility, instantiate a + :class:`~email.generator.BytesGenerator` instance and use its + :meth:`~email.generator.BytesGenerator.flatten` method directly. + For example:: + + from io import BytesIO + from email.generator import BytesGenerator + fp = BytesIO() + g = BytesGenerator(fp, mangle_from_=True, maxheaderlen=60) + g.flatten(msg) + text = fp.getvalue() + + .. versionadded:: 3.4 + + + .. method:: __bytes__() + + Equivalent to :meth:`.as_bytes()`. Allows ``bytes(msg)`` to produce a + bytes object containing the formatted message. + + .. versionadded:: 3.4 + + + .. method:: is_multipart() + + Return ``True`` if the message's payload is a list of sub-\ + :class:`Message` objects, otherwise return ``False``. When + :meth:`is_multipart` returns ``False``, the payload should be a string + object (which might be a CTE encoded binary payload. (Note that + :meth:`is_multipart` returning ``True`` does not necessarily mean that + "msg.get_content_maintype() == 'multipart'" will return the ``True``. + For example, ``is_multipart`` will return ``True`` when the + :class:`Message` is of type ``message/rfc822``.) + + + .. method:: set_unixfrom(unixfrom) + + Set the message's envelope header to *unixfrom*, which should be a string. + + + .. method:: get_unixfrom() + + Return the message's envelope header. Defaults to ``None`` if the + envelope header was never set. + + + .. method:: attach(payload) + + Add the given *payload* to the current payload, which must be ``None`` or + a list of :class:`Message` objects before the call. After the call, the + payload will always be a list of :class:`Message` objects. If you want to + set the payload to a scalar object (e.g. a string), use + :meth:`set_payload` instead. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by :meth:`~email.message.EmailMessage.set_content` and the + related ``make`` and ``add`` methods. + + + .. method:: get_payload(i=None, decode=False) + + Return the current payload, which will be a list of + :class:`Message` objects when :meth:`is_multipart` is ``True``, or a + string when :meth:`is_multipart` is ``False``. If the payload is a list + and you mutate the list object, you modify the message's payload in place. + + With optional argument *i*, :meth:`get_payload` will return the *i*-th + element of the payload, counting from zero, if :meth:`is_multipart` is + ``True``. An :exc:`IndexError` will be raised if *i* is less than 0 or + greater than or equal to the number of items in the payload. If the + payload is a string (i.e. :meth:`is_multipart` is ``False``) and *i* is + given, a :exc:`TypeError` is raised. + + Optional *decode* is a flag indicating whether the payload should be + decoded or not, according to the :mailheader:`Content-Transfer-Encoding` + header. When ``True`` and the message is not a multipart, the payload will + be decoded if this header's value is ``quoted-printable`` or ``base64``. + If some other encoding is used, or :mailheader:`Content-Transfer-Encoding` + header is missing, the payload is + returned as-is (undecoded). In all cases the returned value is binary + data. If the message is a multipart and the *decode* flag is ``True``, + then ``None`` is returned. If the payload is base64 and it was not + perfectly formed (missing padding, characters outside the base64 + alphabet), then an appropriate defect will be added to the message's + defect property (:class:`~email.errors.InvalidBase64PaddingDefect` or + :class:`~email.errors.InvalidBase64CharactersDefect`, respectively). + + When *decode* is ``False`` (the default) the body is returned as a string + without decoding the :mailheader:`Content-Transfer-Encoding`. However, + for a :mailheader:`Content-Transfer-Encoding` of 8bit, an attempt is made + to decode the original bytes using the ``charset`` specified by the + :mailheader:`Content-Type` header, using the ``replace`` error handler. + If no ``charset`` is specified, or if the ``charset`` given is not + recognized by the email package, the body is decoded using the default + ASCII charset. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by :meth:`~email.message.EmailMessage.get_content` and + :meth:`~email.message.EmailMessage.iter_parts`. + + + .. method:: set_payload(payload, charset=None) + + Set the entire message object's payload to *payload*. It is the client's + responsibility to ensure the payload invariants. Optional *charset* sets + the message's default character set; see :meth:`set_charset` for details. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by :meth:`~email.message.EmailMessage.set_content`. + + + .. method:: set_charset(charset) + + Set the character set of the payload to *charset*, which can either be a + :class:`~email.charset.Charset` instance (see :mod:`email.charset`), a + string naming a character set, or ``None``. If it is a string, it will + be converted to a :class:`~email.charset.Charset` instance. If *charset* + is ``None``, the ``charset`` parameter will be removed from the + :mailheader:`Content-Type` header (the message will not be otherwise + modified). Anything else will generate a :exc:`TypeError`. + + If there is no existing :mailheader:`MIME-Version` header one will be + added. If there is no existing :mailheader:`Content-Type` header, one + will be added with a value of :mimetype:`text/plain`. Whether the + :mailheader:`Content-Type` header already exists or not, its ``charset`` + parameter will be set to *charset.output_charset*. If + *charset.input_charset* and *charset.output_charset* differ, the payload + will be re-encoded to the *output_charset*. If there is no existing + :mailheader:`Content-Transfer-Encoding` header, then the payload will be + transfer-encoded, if needed, using the specified + :class:`~email.charset.Charset`, and a header with the appropriate value + will be added. If a :mailheader:`Content-Transfer-Encoding` header + already exists, the payload is assumed to already be correctly encoded + using that :mailheader:`Content-Transfer-Encoding` and is not modified. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the *charset* parameter of the + :meth:`email.emailmessage.EmailMessage.set_content` method. + + + .. method:: get_charset() + + Return the :class:`~email.charset.Charset` instance associated with the + message's payload. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class it always returns + ``None``. + + + The following methods implement a mapping-like interface for accessing the + message's :rfc:`2822` headers. Note that there are some semantic differences + between these methods and a normal mapping (i.e. dictionary) interface. For + example, in a dictionary there are no duplicate keys, but here there may be + duplicate message headers. Also, in dictionaries there is no guaranteed + order to the keys returned by :meth:`keys`, but in a :class:`Message` object, + headers are always returned in the order they appeared in the original + message, or were added to the message later. Any header deleted and then + re-added are always appended to the end of the header list. + + These semantic differences are intentional and are biased toward maximal + convenience. + + Note that in all cases, any envelope header present in the message is not + included in the mapping interface. + + In a model generated from bytes, any header values that (in contravention of + the RFCs) contain non-ASCII bytes will, when retrieved through this + interface, be represented as :class:`~email.header.Header` objects with + a charset of `unknown-8bit`. + + + .. method:: __len__() + + Return the total number of headers, including duplicates. + + + .. method:: __contains__(name) + + Return true if the message object has a field named *name*. Matching is + done case-insensitively and *name* should not include the trailing colon. + Used for the ``in`` operator, e.g.:: + + if 'message-id' in myMessage: + print('Message-ID:', myMessage['message-id']) + + + .. method:: __getitem__(name) + + Return the value of the named header field. *name* should not include the + colon field separator. If the header is missing, ``None`` is returned; a + :exc:`KeyError` is never raised. + + Note that if the named field appears more than once in the message's + headers, exactly which of those field values will be returned is + undefined. Use the :meth:`get_all` method to get the values of all the + extant named headers. + + + .. method:: __setitem__(name, val) + + Add a header to the message with field name *name* and value *val*. The + field is appended to the end of the message's existing fields. + + Note that this does *not* overwrite or delete any existing header with the same + name. If you want to ensure that the new header is the only one present in the + message with field name *name*, delete the field first, e.g.:: + + del msg['subject'] + msg['subject'] = 'Python roolz!' + + + .. method:: __delitem__(name) + + Delete all occurrences of the field with name *name* from the message's + headers. No exception is raised if the named field isn't present in the + headers. + + + .. method:: keys() + + Return a list of all the message's header field names. + + + .. method:: values() + + Return a list of all the message's field values. + + + .. method:: items() + + Return a list of 2-tuples containing all the message's field headers and + values. + + + .. method:: get(name, failobj=None) + + Return the value of the named header field. This is identical to + :meth:`__getitem__` except that optional *failobj* is returned if the + named header is missing (defaults to ``None``). + + Here are some additional useful methods: + + + .. method:: get_all(name, failobj=None) + + Return a list of all the values for the field named *name*. If there are + no such named headers in the message, *failobj* is returned (defaults to + ``None``). + + + .. method:: add_header(_name, _value, **_params) + + Extended header setting. This method is similar to :meth:`__setitem__` + except that additional header parameters can be provided as keyword + arguments. *_name* is the header field to add and *_value* is the + *primary* value for the header. + + For each item in the keyword argument dictionary *_params*, the key is + taken as the parameter name, with underscores converted to dashes (since + dashes are illegal in Python identifiers). Normally, the parameter will + be added as ``key="value"`` unless the value is ``None``, in which case + only the key will be added. If the value contains non-ASCII characters, + it can be specified as a three tuple in the format + ``(CHARSET, LANGUAGE, VALUE)``, where ``CHARSET`` is a string naming the + charset to be used to encode the value, ``LANGUAGE`` can usually be set + to ``None`` or the empty string (see :rfc:`2231` for other possibilities), + and ``VALUE`` is the string value containing non-ASCII code points. If + a three tuple is not passed and the value contains non-ASCII characters, + it is automatically encoded in :rfc:`2231` format using a ``CHARSET`` + of ``utf-8`` and a ``LANGUAGE`` of ``None``. + + Here's an example:: + + msg.add_header('Content-Disposition', 'attachment', filename='bud.gif') + + This will add a header that looks like :: + + Content-Disposition: attachment; filename="bud.gif" + + An example with non-ASCII characters:: + + msg.add_header('Content-Disposition', 'attachment', + filename=('iso-8859-1', '', 'Fußballer.ppt')) + + Which produces :: + + Content-Disposition: attachment; filename*="iso-8859-1''Fu%DFballer.ppt" + + + .. method:: replace_header(_name, _value) + + Replace a header. Replace the first header found in the message that + matches *_name*, retaining header order and field name case. If no + matching header was found, a :exc:`KeyError` is raised. + + + .. method:: get_content_type() + + Return the message's content type. The returned string is coerced to + lower case of the form :mimetype:`maintype/subtype`. If there was no + :mailheader:`Content-Type` header in the message the default type as given + by :meth:`get_default_type` will be returned. Since according to + :rfc:`2045`, messages always have a default type, :meth:`get_content_type` + will always return a value. + + :rfc:`2045` defines a message's default type to be :mimetype:`text/plain` + unless it appears inside a :mimetype:`multipart/digest` container, in + which case it would be :mimetype:`message/rfc822`. If the + :mailheader:`Content-Type` header has an invalid type specification, + :rfc:`2045` mandates that the default type be :mimetype:`text/plain`. + + + .. method:: get_content_maintype() + + Return the message's main content type. This is the :mimetype:`maintype` + part of the string returned by :meth:`get_content_type`. + + + .. method:: get_content_subtype() + + Return the message's sub-content type. This is the :mimetype:`subtype` + part of the string returned by :meth:`get_content_type`. + + + .. method:: get_default_type() + + Return the default content type. Most messages have a default content + type of :mimetype:`text/plain`, except for messages that are subparts of + :mimetype:`multipart/digest` containers. Such subparts have a default + content type of :mimetype:`message/rfc822`. + + + .. method:: set_default_type(ctype) + + Set the default content type. *ctype* should either be + :mimetype:`text/plain` or :mimetype:`message/rfc822`, although this is not + enforced. The default content type is not stored in the + :mailheader:`Content-Type` header. + + + .. method:: get_params(failobj=None, header='content-type', unquote=True) + + Return the message's :mailheader:`Content-Type` parameters, as a list. + The elements of the returned list are 2-tuples of key/value pairs, as + split on the ``'='`` sign. The left hand side of the ``'='`` is the key, + while the right hand side is the value. If there is no ``'='`` sign in + the parameter the value is the empty string, otherwise the value is as + described in :meth:`get_param` and is unquoted if optional *unquote* is + ``True`` (the default). + + Optional *failobj* is the object to return if there is no + :mailheader:`Content-Type` header. Optional *header* is the header to + search instead of :mailheader:`Content-Type`. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the *params* property of the individual header objects + returned by the header access methods. + + + .. method:: get_param(param, failobj=None, header='content-type', unquote=True) + + Return the value of the :mailheader:`Content-Type` header's parameter + *param* as a string. If the message has no :mailheader:`Content-Type` + header or if there is no such parameter, then *failobj* is returned + (defaults to ``None``). + + Optional *header* if given, specifies the message header to use instead of + :mailheader:`Content-Type`. + + Parameter keys are always compared case insensitively. The return value + can either be a string, or a 3-tuple if the parameter was :rfc:`2231` + encoded. When it's a 3-tuple, the elements of the value are of the form + ``(CHARSET, LANGUAGE, VALUE)``. Note that both ``CHARSET`` and + ``LANGUAGE`` can be ``None``, in which case you should consider ``VALUE`` + to be encoded in the ``us-ascii`` charset. You can usually ignore + ``LANGUAGE``. + + If your application doesn't care whether the parameter was encoded as in + :rfc:`2231`, you can collapse the parameter value by calling + :func:`email.utils.collapse_rfc2231_value`, passing in the return value + from :meth:`get_param`. This will return a suitably decoded Unicode + string when the value is a tuple, or the original string unquoted if it + isn't. For example:: + + rawparam = msg.get_param('foo') + param = email.utils.collapse_rfc2231_value(rawparam) + + In any case, the parameter value (either the returned string, or the + ``VALUE`` item in the 3-tuple) is always unquoted, unless *unquote* is set + to ``False``. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the *params* property of the individual header objects + returned by the header access methods. + + + .. method:: set_param(param, value, header='Content-Type', requote=True, \ + charset=None, language='', replace=False) + + Set a parameter in the :mailheader:`Content-Type` header. If the + parameter already exists in the header, its value will be replaced with + *value*. If the :mailheader:`Content-Type` header as not yet been defined + for this message, it will be set to :mimetype:`text/plain` and the new + parameter value will be appended as per :rfc:`2045`. + + Optional *header* specifies an alternative header to + :mailheader:`Content-Type`, and all parameters will be quoted as necessary + unless optional *requote* is ``False`` (the default is ``True``). + + If optional *charset* is specified, the parameter will be encoded + according to :rfc:`2231`. Optional *language* specifies the RFC 2231 + language, defaulting to the empty string. Both *charset* and *language* + should be strings. + + If *replace* is ``False`` (the default) the header is moved to the + end of the list of headers. If *replace* is ``True``, the header + will be updated in place. + + .. versionchanged:: 3.4 ``replace`` keyword was added. + + + .. method:: del_param(param, header='content-type', requote=True) + + Remove the given parameter completely from the :mailheader:`Content-Type` + header. The header will be re-written in place without the parameter or + its value. All values will be quoted as necessary unless *requote* is + ``False`` (the default is ``True``). Optional *header* specifies an + alternative to :mailheader:`Content-Type`. + + + .. method:: set_type(type, header='Content-Type', requote=True) + + Set the main type and subtype for the :mailheader:`Content-Type` + header. *type* must be a string in the form :mimetype:`maintype/subtype`, + otherwise a :exc:`ValueError` is raised. + + This method replaces the :mailheader:`Content-Type` header, keeping all + the parameters in place. If *requote* is ``False``, this leaves the + existing header's quoting as is, otherwise the parameters will be quoted + (the default). + + An alternative header can be specified in the *header* argument. When the + :mailheader:`Content-Type` header is set a :mailheader:`MIME-Version` + header is also added. + + This is a legacy method. On the + :class:`~email.emailmessage.EmailMessage` class its functionality is + replaced by the ``make_`` and ``add_`` methods. + + + .. method:: get_filename(failobj=None) + + Return the value of the ``filename`` parameter of the + :mailheader:`Content-Disposition` header of the message. If the header + does not have a ``filename`` parameter, this method falls back to looking + for the ``name`` parameter on the :mailheader:`Content-Type` header. If + neither is found, or the header is missing, then *failobj* is returned. + The returned string will always be unquoted as per + :func:`email.utils.unquote`. + + + .. method:: get_boundary(failobj=None) + + Return the value of the ``boundary`` parameter of the + :mailheader:`Content-Type` header of the message, or *failobj* if either + the header is missing, or has no ``boundary`` parameter. The returned + string will always be unquoted as per :func:`email.utils.unquote`. + + + .. method:: set_boundary(boundary) + + Set the ``boundary`` parameter of the :mailheader:`Content-Type` header to + *boundary*. :meth:`set_boundary` will always quote *boundary* if + necessary. A :exc:`~email.errors.HeaderParseError` is raised if the + message object has no :mailheader:`Content-Type` header. + + Note that using this method is subtly different than deleting the old + :mailheader:`Content-Type` header and adding a new one with the new + boundary via :meth:`add_header`, because :meth:`set_boundary` preserves + the order of the :mailheader:`Content-Type` header in the list of + headers. However, it does *not* preserve any continuation lines which may + have been present in the original :mailheader:`Content-Type` header. + + + .. method:: get_content_charset(failobj=None) + + Return the ``charset`` parameter of the :mailheader:`Content-Type` header, + coerced to lower case. If there is no :mailheader:`Content-Type` header, or if + that header has no ``charset`` parameter, *failobj* is returned. + + Note that this method differs from :meth:`get_charset` which returns the + :class:`~email.charset.Charset` instance for the default encoding of the message body. + + + .. method:: get_charsets(failobj=None) + + Return a list containing the character set names in the message. If the + message is a :mimetype:`multipart`, then the list will contain one element + for each subpart in the payload, otherwise, it will be a list of length 1. + + Each item in the list will be a string which is the value of the + ``charset`` parameter in the :mailheader:`Content-Type` header for the + represented subpart. However, if the subpart has no + :mailheader:`Content-Type` header, no ``charset`` parameter, or is not of + the :mimetype:`text` main MIME type, then that item in the returned list + will be *failobj*. + + + .. method:: get_content_disposition() + + Return the lowercased value (without parameters) of the message's + :mailheader:`Content-Disposition` header if it has one, or ``None``. The + possible values for this method are *inline*, *attachment* or ``None`` + if the message follows :rfc:`2183`. + + .. versionadded:: 3.5 + + .. method:: walk() + + The :meth:`walk` method is an all-purpose generator which can be used to + iterate over all the parts and subparts of a message object tree, in + depth-first traversal order. You will typically use :meth:`walk` as the + iterator in a ``for`` loop; each iteration returns the next subpart. + + Here's an example that prints the MIME type of every part of a multipart + message structure: + + .. testsetup:: + + import email + from email import message_from_binary_file + from os.path import join, dirname + lib_dir = dirname(dirname(email.__file__)) + file_path = join(lib_dir, 'test/test_email/data/msg_16.txt') + with open(file_path, 'rb') as f: + msg = message_from_binary_file(f) + from email.iterators import _structure + + .. doctest:: + + >>> for part in msg.walk(): + ... print(part.get_content_type()) + multipart/report + text/plain + message/delivery-status + text/plain + text/plain + message/rfc822 + text/plain + + ``walk`` iterates over the subparts of any part where + :meth:`is_multipart` returns ``True``, even though + ``msg.get_content_maintype() == 'multipart'`` may return ``False``. We + can see this in our example by making use of the ``_structure`` debug + helper function: + + .. doctest:: + + >>> for part in msg.walk(): + ... print(part.get_content_maintype() == 'multipart', + ... part.is_multipart()) + True True + False False + False True + False False + False False + False True + False False + >>> _structure(msg) + multipart/report + text/plain + message/delivery-status + text/plain + text/plain + message/rfc822 + text/plain + + Here the ``message`` parts are not ``multiparts``, but they do contain + subparts. ``is_multipart()`` returns ``True`` and ``walk`` descends + into the subparts. + + + :class:`Message` objects can also optionally contain two instance attributes, + which can be used when generating the plain text of a MIME message. + + + .. attribute:: preamble + + The format of a MIME document allows for some text between the blank line + following the headers, and the first multipart boundary string. Normally, + this text is never visible in a MIME-aware mail reader because it falls + outside the standard MIME armor. However, when viewing the raw text of + the message, or when viewing the message in a non-MIME aware reader, this + text can become visible. + + The *preamble* attribute contains this leading extra-armor text for MIME + documents. When the :class:`~email.parser.Parser` discovers some text + after the headers but before the first boundary string, it assigns this + text to the message's *preamble* attribute. When the + :class:`~email.generator.Generator` is writing out the plain text + representation of a MIME message, and it finds the + message has a *preamble* attribute, it will write this text in the area + between the headers and the first boundary. See :mod:`email.parser` and + :mod:`email.generator` for details. + + Note that if the message object has no preamble, the *preamble* attribute + will be ``None``. + + + .. attribute:: epilogue + + The *epilogue* attribute acts the same way as the *preamble* attribute, + except that it contains text that appears between the last boundary and + the end of the message. + + You do not need to set the epilogue to the empty string in order for the + :class:`~email.generator.Generator` to print a newline at the end of the + file. + + + .. attribute:: defects + + The *defects* attribute contains a list of all the problems found when + parsing this message. See :mod:`email.errors` for a detailed description + of the possible parsing defects. diff --git a/Doc/library/email.contentmanager.rst b/Doc/library/email.contentmanager.rst new file mode 100644 index 00000000000000..f56836ae2d2781 --- /dev/null +++ b/Doc/library/email.contentmanager.rst @@ -0,0 +1,207 @@ +:mod:`email.contentmanager`: Managing MIME Content +-------------------------------------------------- + +.. module:: email.contentmanager + :synopsis: Storing and Retrieving Content from MIME Parts + +.. moduleauthor:: R. David Murray +.. sectionauthor:: R. David Murray + +**Source code:** :source:`Lib/email/contentmanager.py` + +------------ + +.. versionadded:: 3.6 [1]_ + + +.. class:: ContentManager() + + Base class for content managers. Provides the standard registry mechanisms + to register converters between MIME content and other representations, as + well as the ``get_content`` and ``set_content`` dispatch methods. + + + .. method:: get_content(msg, *args, **kw) + + Look up a handler function based on the ``mimetype`` of *msg* (see next + paragraph), call it, passing through all arguments, and return the result + of the call. The expectation is that the handler will extract the + payload from *msg* and return an object that encodes information about + the extracted data. + + To find the handler, look for the following keys in the registry, + stopping with the first one found: + + * the string representing the full MIME type (``maintype/subtype``) + * the string representing the ``maintype`` + * the empty string + + If none of these keys produce a handler, raise a :exc:`KeyError` for the + full MIME type. + + + .. method:: set_content(msg, obj, *args, **kw) + + If the ``maintype`` is ``multipart``, raise a :exc:`TypeError`; otherwise + look up a handler function based on the type of *obj* (see next + paragraph), call :meth:`~email.message.EmailMessage.clear_content` on the + *msg*, and call the handler function, passing through all arguments. The + expectation is that the handler will transform and store *obj* into + *msg*, possibly making other changes to *msg* as well, such as adding + various MIME headers to encode information needed to interpret the stored + data. + + To find the handler, obtain the type of *obj* (``typ = type(obj)``), and + look for the following keys in the registry, stopping with the first one + found: + + * the type itself (``typ``) + * the type's fully qualified name (``typ.__module__ + '.' + + typ.__qualname__``). + * the type's qualname (``typ.__qualname__``) + * the type's name (``typ.__name__``). + + If none of the above match, repeat all of the checks above for each of + the types in the :term:`MRO` (``typ.__mro__``). Finally, if no other key + yields a handler, check for a handler for the key ``None``. If there is + no handler for ``None``, raise a :exc:`KeyError` for the fully + qualified name of the type. + + Also add a :mailheader:`MIME-Version` header if one is not present (see + also :class:`.MIMEPart`). + + + .. method:: add_get_handler(key, handler) + + Record the function *handler* as the handler for *key*. For the possible + values of *key*, see :meth:`get_content`. + + + .. method:: add_set_handler(typekey, handler) + + Record *handler* as the function to call when an object of a type + matching *typekey* is passed to :meth:`set_content`. For the possible + values of *typekey*, see :meth:`set_content`. + + +Content Manager Instances +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently the email package provides only one concrete content manager, +:data:`raw_data_manager`, although more may be added in the future. +:data:`raw_data_manager` is the +:attr:`~email.policy.EmailPolicy.content_manager` provided by +:attr:`~email.policy.EmailPolicy` and its derivatives. + + +.. data:: raw_data_manager + + This content manager provides only a minimum interface beyond that provided + by :class:`~email.message.Message` itself: it deals only with text, raw + byte strings, and :class:`~email.message.Message` objects. Nevertheless, it + provides significant advantages compared to the base API: ``get_content`` on + a text part will return a unicode string without the application needing to + manually decode it, ``set_content`` provides a rich set of options for + controlling the headers added to a part and controlling the content transfer + encoding, and it enables the use of the various ``add_`` methods, thereby + simplifying the creation of multipart messages. + + .. method:: get_content(msg, errors='replace') + + Return the payload of the part as either a string (for ``text`` parts), an + :class:`~email.message.EmailMessage` object (for ``message/rfc822`` + parts), or a ``bytes`` object (for all other non-multipart types). Raise + a :exc:`KeyError` if called on a ``multipart``. If the part is a + ``text`` part and *errors* is specified, use it as the error handler when + decoding the payload to unicode. The default error handler is + ``replace``. + + .. method:: set_content(msg, <'str'>, subtype="plain", charset='utf-8' \ + cte=None, \ + disposition=None, filename=None, cid=None, \ + params=None, headers=None) + set_content(msg, <'bytes'>, maintype, subtype, cte="base64", \ + disposition=None, filename=None, cid=None, \ + params=None, headers=None) + set_content(msg, <'EmailMessage'>, cte=None, \ + disposition=None, filename=None, cid=None, \ + params=None, headers=None) + set_content(msg, <'list'>, subtype='mixed', \ + disposition=None, filename=None, cid=None, \ + params=None, headers=None) + + Add headers and payload to *msg*: + + Add a :mailheader:`Content-Type` header with a ``maintype/subtype`` + value. + + * For ``str``, set the MIME ``maintype`` to ``text``, and set the + subtype to *subtype* if it is specified, or ``plain`` if it is not. + * For ``bytes``, use the specified *maintype* and *subtype*, or + raise a :exc:`TypeError` if they are not specified. + * For :class:`~email.message.EmailMessage` objects, set the maintype + to ``message``, and set the subtype to *subtype* if it is + specified or ``rfc822`` if it is not. If *subtype* is + ``partial``, raise an error (``bytes`` objects must be used to + construct ``message/partial`` parts). + * For *<'list'>*, which should be a list of + :class:`~email.message.EmailMessage` objects, set the ``maintype`` + to ``multipart``, and the ``subtype`` to *subtype* if it is + specified, and ``mixed`` if it is not. If the message parts in + the *<'list'>* have :mailheader:`MIME-Version` headers, remove + them. + + If *charset* is provided (which is valid only for ``str``), encode the + string to bytes using the specified character set. The default is + ``utf-8``. If the specified *charset* is a known alias for a standard + MIME charset name, use the standard charset instead. + + If *cte* is set, encode the payload using the specified content transfer + encoding, and set the :mailheader:`Content-Transfer-Encoding` header to + that value. Possible values for *cte* are ``quoted-printable``, + ``base64``, ``7bit``, ``8bit``, and ``binary``. If the input cannot be + encoded in the specified encoding (for example, specifying a *cte* of + ``7bit`` for an input that contains non-ASCII values), raise a + :exc:`ValueError`. + + * For ``str`` objects, if *cte* is not set use heuristics to + determine the most compact encoding. + * For :class:`~email.message.EmailMessage`, per :rfc:`2046`, raise + an error if a *cte* of ``quoted-printable`` or ``base64`` is + requested for *subtype* ``rfc822``, and for any *cte* other than + ``7bit`` for *subtype* ``external-body``. For + ``message/rfc822``, use ``8bit`` if *cte* is not specified. For + all other values of *subtype*, use ``7bit``. + + .. note:: A *cte* of ``binary`` does not actually work correctly yet. + The ``EmailMessage`` object as modified by ``set_content`` is + correct, but :class:`~email.generator.BytesGenerator` does not + serialize it correctly. + + If *disposition* is set, use it as the value of the + :mailheader:`Content-Disposition` header. If not specified, and + *filename* is specified, add the header with the value ``attachment``. + If *disposition* is not specified and *filename* is also not specified, + do not add the header. The only valid values for *disposition* are + ``attachment`` and ``inline``. + + If *filename* is specified, use it as the value of the ``filename`` + parameter of the :mailheader:`Content-Disposition` header. + + If *cid* is specified, add a :mailheader:`Content-ID` header with + *cid* as its value. + + If *params* is specified, iterate its ``items`` method and use the + resulting ``(key, value)`` pairs to set additional parameters on the + :mailheader:`Content-Type` header. + + If *headers* is specified and is a list of strings of the form + ``headername: headervalue`` or a list of ``header`` objects + (distinguished from strings by having a ``name`` attribute), add the + headers to *msg*. + + +.. rubric:: Footnotes + +.. [1] Originally added in 3.4 as a :term:`provisional module ` diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst index 499d754fa08958..f20cdea67a43b1 100644 --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -93,3 +93,13 @@ this class is *not* an exception! has this defect, its :meth:`~email.message.Message.is_multipart` method may return false even though its content type claims to be :mimetype:`multipart`. +<<<<<<< HEAD +======= +* :class:`InvalidBase64PaddingDefect` -- When decoding a block of base64 + encoded bytes, the padding was not correct. Enough padding is added to + perform the decode, but the resulting decoded bytes may be invalid. + +* :class:`InvalidBase64CharactersDefect` -- When decoding a block of base64 + encoded bytes, characters outside the base64 alphabet were encountered. + The characters are ignored, but the resulting decoded bytes may be invalid. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst index 8fe70a7b926de7..661fafce83d649 100644 --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -4,6 +4,107 @@ .. module:: email.generator :synopsis: Generate flat text email messages from a message structure. +<<<<<<< HEAD +======= +**Source code:** :source:`Lib/email/generator.py` + +-------------- + +One of the most common tasks is to generate the flat (serialized) version of +the email message represented by a message object structure. You will need to +do this if you want to send your message via :meth:`smtplib.SMTP.sendmail` or +the :mod:`nntplib` module, or print the message on the console. Taking a +message object structure and producing a serialized representation is the job +of the generator classes. + +As with the :mod:`email.parser` module, you aren't limited to the functionality +of the bundled generator; you could write one from scratch yourself. However +the bundled generator knows how to generate most email in a standards-compliant +way, should handle MIME and non-MIME email messages just fine, and is designed +so that the bytes-oriented parsing and generation operations are inverses, +assuming the same non-transforming :mod:`~email.policy` is used for both. That +is, parsing the serialized byte stream via the +:class:`~email.parser.BytesParser` class and then regenerating the serialized +byte stream using :class:`BytesGenerator` should produce output identical to +the input [#]_. (On the other hand, using the generator on an +:class:`~email.message.EmailMessage` constructed by program may result in +changes to the :class:`~email.message.EmailMessage` object as defaults are +filled in.) + +The :class:`Generator` class can be used to flatten a message into a text (as +opposed to binary) serialized representation, but since Unicode cannot +represent binary data directly, the message is of necessity transformed into +something that contains only ASCII characters, using the standard email RFC +Content Transfer Encoding techniques for encoding email messages for transport +over channels that are not "8 bit clean". + + +.. class:: BytesGenerator(outfp, mangle_from_=None, maxheaderlen=None, *, \ + policy=None) + + Return a :class:`BytesGenerator` object that will write any message provided + to the :meth:`flatten` method, or any surrogateescape encoded text provided + to the :meth:`write` method, to the :term:`file-like object` *outfp*. + *outfp* must support a ``write`` method that accepts binary data. + + If optional *mangle_from_* is ``True``, put a ``>`` character in front of + any line in the body that starts with the exact string ``"From "``, that is + ``From`` followed by a space at the beginning of a line. *mangle_from_* + defaults to the value of the :attr:`~email.policy.Policy.mangle_from_` + setting of the *policy* (which is ``True`` for the + :data:`~email.policy.compat32` policy and ``False`` for all others). + *mangle_from_* is intended for use when messages are stored in unix mbox + format (see :mod:`mailbox` and `WHY THE CONTENT-LENGTH FORMAT IS BAD + `_). + + If *maxheaderlen* is not ``None``, refold any header lines that are longer + than *maxheaderlen*, or if ``0``, do not rewrap any headers. If + *manheaderlen* is ``None`` (the default), wrap headers and other message + lines according to the *policy* settings. + + If *policy* is specified, use that policy to control message generation. If + *policy* is ``None`` (the default), use the policy associated with the + :class:`~email.message.Message` or :class:`~email.message.EmailMessage` + object passed to ``flatten`` to control the message generation. See + :mod:`email.policy` for details on what *policy* controls. + + .. versionadded:: 3.2 + + .. versionchanged:: 3.3 Added the *policy* keyword. + + .. versionchanged:: 3.6 The default behavior of the *mangle_from_* + and *maxheaderlen* parameters is to follow the policy. + + + .. method:: flatten(msg, unixfrom=False, linesep=None) + + Print the textual representation of the message object structure rooted + at *msg* to the output file specified when the :class:`BytesGenerator` + instance was created. + + If the :mod:`~email.policy` option :attr:`~email.policy.Policy.cte_type` + is ``8bit`` (the default), copy any headers in the original parsed + message that have not been modified to the output with any bytes with the + high bit set reproduced as in the original, and preserve the non-ASCII + :mailheader:`Content-Transfer-Encoding` of any body parts that have them. + If ``cte_type`` is ``7bit``, convert the bytes with the high bit set as + needed using an ASCII-compatible :mailheader:`Content-Transfer-Encoding`. + That is, transform parts with non-ASCII + :mailheader:`Content-Transfer-Encoding` + (:mailheader:`Content-Transfer-Encoding: 8bit`) to an ASCII compatible + :mailheader:`Content-Transfer-Encoding`, and encode RFC-invalid non-ASCII + bytes in headers using the MIME ``unknown-8bit`` character set, thus + rendering them RFC-compliant. + + .. XXX: There should be an option that just does the RFC + compliance transformation on headers but leaves CTE 8bit parts alone. + + If *unixfrom* is ``True``, print the envelope header delimiter used by + the Unix mailbox format (see :mod:`mailbox`) before the first of the + :rfc:`5322` headers of the root message object. If the root object has + no envelope header, craft a standard one. The default is ``False``. + Note that for subparts, no envelope header is ever printed. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) One of the most common tasks is to generate the flat text of the email message represented by a message object structure. You will need to do this if you want diff --git a/Doc/library/email.headerregistry.rst b/Doc/library/email.headerregistry.rst new file mode 100644 index 00000000000000..ce283c6b596cf2 --- /dev/null +++ b/Doc/library/email.headerregistry.rst @@ -0,0 +1,455 @@ +:mod:`email.headerregistry`: Custom Header Objects +-------------------------------------------------- + +.. module:: email.headerregistry + :synopsis: Automatic Parsing of headers based on the field name + +.. moduleauthor:: R. David Murray +.. sectionauthor:: R. David Murray + +**Source code:** :source:`Lib/email/headerregistry.py` + +-------------- + +.. versionadded:: 3.6 [1]_ + +Headers are represented by customized subclasses of :class:`str`. The +particular class used to represent a given header is determined by the +:attr:`~email.policy.EmailPolicy.header_factory` of the :mod:`~email.policy` in +effect when the headers are created. This section documents the particular +``header_factory`` implemented by the email package for handling :RFC:`5322` +compliant email messages, which not only provides customized header objects for +various header types, but also provides an extension mechanism for applications +to add their own custom header types. + +When using any of the policy objects derived from +:data:`~email.policy.EmailPolicy`, all headers are produced by +:class:`.HeaderRegistry` and have :class:`.BaseHeader` as their last base +class. Each header class has an additional base class that is determined by +the type of the header. For example, many headers have the class +:class:`.UnstructuredHeader` as their other base class. The specialized second +class for a header is determined by the name of the header, using a lookup +table stored in the :class:`.HeaderRegistry`. All of this is managed +transparently for the typical application program, but interfaces are provided +for modifying the default behavior for use by more complex applications. + +The sections below first document the header base classes and their attributes, +followed by the API for modifying the behavior of :class:`.HeaderRegistry`, and +finally the support classes used to represent the data parsed from structured +headers. + + +.. class:: BaseHeader(name, value) + + *name* and *value* are passed to ``BaseHeader`` from the + :attr:`~email.policy.EmailPolicy.header_factory` call. The string value of + any header object is the *value* fully decoded to unicode. + + This base class defines the following read-only properties: + + + .. attribute:: name + + The name of the header (the portion of the field before the ':'). This + is exactly the value passed in the + :attr:`~email.policy.EmailPolicy.header_factory` call for *name*; that + is, case is preserved. + + + .. attribute:: defects + + A tuple of :exc:`~email.errors.HeaderDefect` instances reporting any + RFC compliance problems found during parsing. The email package tries to + be complete about detecting compliance issues. See the :mod:`~email.errors` + module for a discussion of the types of defects that may be reported. + + + .. attribute:: max_count + + The maximum number of headers of this type that can have the same + ``name``. A value of ``None`` means unlimited. The ``BaseHeader`` value + for this attribute is ``None``; it is expected that specialized header + classes will override this value as needed. + + ``BaseHeader`` also provides the following method, which is called by the + email library code and should not in general be called by application + programs: + + .. method:: fold(*, policy) + + Return a string containing :attr:`~email.policy.Policy.linesep` + characters as required to correctly fold the header according to + *policy*. A :attr:`~email.policy.Policy.cte_type` of ``8bit`` will be + treated as if it were ``7bit``, since headers may not contain arbitrary + binary data. If :attr:`~email.policy.EmailPolicy.utf8` is ``False``, + non-ASCII data will be :rfc:`2047` encoded. + + + ``BaseHeader`` by itself cannot be used to create a header object. It + defines a protocol that each specialized header cooperates with in order to + produce the header object. Specifically, ``BaseHeader`` requires that + the specialized class provide a :func:`classmethod` named ``parse``. This + method is called as follows:: + + parse(string, kwds) + + ``kwds`` is a dictionary containing one pre-initialized key, ``defects``. + ``defects`` is an empty list. The parse method should append any detected + defects to this list. On return, the ``kwds`` dictionary *must* contain + values for at least the keys ``decoded`` and ``defects``. ``decoded`` + should be the string value for the header (that is, the header value fully + decoded to unicode). The parse method should assume that *string* may + contain content-transfer-encoded parts, but should correctly handle all valid + unicode characters as well so that it can parse un-encoded header values. + + ``BaseHeader``'s ``__new__`` then creates the header instance, and calls its + ``init`` method. The specialized class only needs to provide an ``init`` + method if it wishes to set additional attributes beyond those provided by + ``BaseHeader`` itself. Such an ``init`` method should look like this:: + + def init(self, *args, **kw): + self._myattr = kw.pop('myattr') + super().init(*args, **kw) + + That is, anything extra that the specialized class puts in to the ``kwds`` + dictionary should be removed and handled, and the remaining contents of + ``kw`` (and ``args``) passed to the ``BaseHeader`` ``init`` method. + + +.. class:: UnstructuredHeader + + An "unstructured" header is the default type of header in :rfc:`5322`. + Any header that does not have a specified syntax is treated as + unstructured. The classic example of an unstructured header is the + :mailheader:`Subject` header. + + In :rfc:`5322`, an unstructured header is a run of arbitrary text in the + ASCII character set. :rfc:`2047`, however, has an :rfc:`5322` compatible + mechanism for encoding non-ASCII text as ASCII characters within a header + value. When a *value* containing encoded words is passed to the + constructor, the ``UnstructuredHeader`` parser converts such encoded words + into unicode, following the :rfc:`2047` rules for unstructured text. The + parser uses heuristics to attempt to decode certain non-compliant encoded + words. Defects are registered in such cases, as well as defects for issues + such as invalid characters within the encoded words or the non-encoded text. + + This header type provides no additional attributes. + + +.. class:: DateHeader + + :rfc:`5322` specifies a very specific format for dates within email headers. + The ``DateHeader`` parser recognizes that date format, as well as + recognizing a number of variant forms that are sometimes found "in the + wild". + + This header type provides the following additional attributes: + + .. attribute:: datetime + + If the header value can be recognized as a valid date of one form or + another, this attribute will contain a :class:`~datetime.datetime` + instance representing that date. If the timezone of the input date is + specified as ``-0000`` (indicating it is in UTC but contains no + information about the source timezone), then :attr:`.datetime` will be a + naive :class:`~datetime.datetime`. If a specific timezone offset is + found (including `+0000`), then :attr:`.datetime` will contain an aware + ``datetime`` that uses :class:`datetime.timezone` to record the timezone + offset. + + The ``decoded`` value of the header is determined by formatting the + ``datetime`` according to the :rfc:`5322` rules; that is, it is set to:: + + email.utils.format_datetime(self.datetime) + + When creating a ``DateHeader``, *value* may be + :class:`~datetime.datetime` instance. This means, for example, that + the following code is valid and does what one would expect:: + + msg['Date'] = datetime(2011, 7, 15, 21) + + Because this is a naive ``datetime`` it will be interpreted as a UTC + timestamp, and the resulting value will have a timezone of ``-0000``. Much + more useful is to use the :func:`~email.utils.localtime` function from the + :mod:`~email.utils` module:: + + msg['Date'] = utils.localtime() + + This example sets the date header to the current time and date using + the current timezone offset. + + +.. class:: AddressHeader + + Address headers are one of the most complex structured header types. + The ``AddressHeader`` class provides a generic interface to any address + header. + + This header type provides the following additional attributes: + + + .. attribute:: groups + + A tuple of :class:`.Group` objects encoding the + addresses and groups found in the header value. Addresses that are + not part of a group are represented in this list as single-address + ``Groups`` whose :attr:`~.Group.display_name` is ``None``. + + + .. attribute:: addresses + + A tuple of :class:`.Address` objects encoding all + of the individual addresses from the header value. If the header value + contains any groups, the individual addresses from the group are included + in the list at the point where the group occurs in the value (that is, + the list of addresses is "flattened" into a one dimensional list). + + The ``decoded`` value of the header will have all encoded words decoded to + unicode. :class:`~encodings.idna` encoded domain names are also decoded to + unicode. The ``decoded`` value is set by :attr:`~str.join`\ ing the + :class:`str` value of the elements of the ``groups`` attribute with ``', + '``. + + A list of :class:`.Address` and :class:`.Group` objects in any combination + may be used to set the value of an address header. ``Group`` objects whose + ``display_name`` is ``None`` will be interpreted as single addresses, which + allows an address list to be copied with groups intact by using the list + obtained from the ``groups`` attribute of the source header. + + +.. class:: SingleAddressHeader + + A subclass of :class:`.AddressHeader` that adds one + additional attribute: + + + .. attribute:: address + + The single address encoded by the header value. If the header value + actually contains more than one address (which would be a violation of + the RFC under the default :mod:`~email.policy`), accessing this attribute + will result in a :exc:`ValueError`. + + +Many of the above classes also have a ``Unique`` variant (for example, +``UniqueUnstructuredHeader``). The only difference is that in the ``Unique`` +variant, :attr:`~.BaseHeader.max_count` is set to 1. + + +.. class:: MIMEVersionHeader + + There is really only one valid value for the :mailheader:`MIME-Version` + header, and that is ``1.0``. For future proofing, this header class + supports other valid version numbers. If a version number has a valid value + per :rfc:`2045`, then the header object will have non-``None`` values for + the following attributes: + + .. attribute:: version + + The version number as a string, with any whitespace and/or comments + removed. + + .. attribute:: major + + The major version number as an integer + + .. attribute:: minor + + The minor version number as an integer + + +.. class:: ParameterizedMIMEHeader + + MIME headers all start with the prefix 'Content-'. Each specific header has + a certain value, described under the class for that header. Some can + also take a list of supplemental parameters, which have a common format. + This class serves as a base for all the MIME headers that take parameters. + + .. attribute:: params + + A dictionary mapping parameter names to parameter values. + + +.. class:: ContentTypeHeader + + A :class:`ParameterizedMIMEHeader` class that handles the + :mailheader:`Content-Type` header. + + .. attribute:: content_type + + The content type string, in the form ``maintype/subtype``. + + .. attribute:: maintype + + .. attribute:: subtype + + +.. class:: ContentDispositionHeader + + A :class:`ParameterizedMIMEHeader` class that handles the + :mailheader:`Content-Disposition` header. + + .. attribute:: content-disposition + + ``inline`` and ``attachment`` are the only valid values in common use. + + +.. class:: ContentTransferEncoding + + Handles the :mailheader:`Content-Transfer-Encoding` header. + + .. attribute:: cte + + Valid values are ``7bit``, ``8bit``, ``base64``, and + ``quoted-printable``. See :rfc:`2045` for more information. + + + +.. class:: HeaderRegistry(base_class=BaseHeader, \ + default_class=UnstructuredHeader, \ + use_default_map=True) + + This is the factory used by :class:`~email.policy.EmailPolicy` by default. + ``HeaderRegistry`` builds the class used to create a header instance + dynamically, using *base_class* and a specialized class retrieved from a + registry that it holds. When a given header name does not appear in the + registry, the class specified by *default_class* is used as the specialized + class. When *use_default_map* is ``True`` (the default), the standard + mapping of header names to classes is copied in to the registry during + initialization. *base_class* is always the last class in the generated + class's ``__bases__`` list. + + The default mappings are: + + :subject: UniqueUnstructuredHeader + :date: UniqueDateHeader + :resent-date: DateHeader + :orig-date: UniqueDateHeader + :sender: UniqueSingleAddressHeader + :resent-sender: SingleAddressHeader + :to: UniqueAddressHeader + :resent-to: AddressHeader + :cc: UniqueAddressHeader + :resent-cc: AddressHeader + :from: UniqueAddressHeader + :resent-from: AddressHeader + :reply-to: UniqueAddressHeader + + ``HeaderRegistry`` has the following methods: + + + .. method:: map_to_type(self, name, cls) + + *name* is the name of the header to be mapped. It will be converted to + lower case in the registry. *cls* is the specialized class to be used, + along with *base_class*, to create the class used to instantiate headers + that match *name*. + + + .. method:: __getitem__(name) + + Construct and return a class to handle creating a *name* header. + + + .. method:: __call__(name, value) + + Retrieves the specialized header associated with *name* from the + registry (using *default_class* if *name* does not appear in the + registry) and composes it with *base_class* to produce a class, + calls the constructed class's constructor, passing it the same + argument list, and finally returns the class instance created thereby. + + +The following classes are the classes used to represent data parsed from +structured headers and can, in general, be used by an application program to +construct structured values to assign to specific headers. + + +.. class:: Address(display_name='', username='', domain='', addr_spec=None) + + The class used to represent an email address. The general form of an + address is:: + + [display_name] + + or:: + + username@domain + + where each part must conform to specific syntax rules spelled out in + :rfc:`5322`. + + As a convenience *addr_spec* can be specified instead of *username* and + *domain*, in which case *username* and *domain* will be parsed from the + *addr_spec*. An *addr_spec* must be a properly RFC quoted string; if it is + not ``Address`` will raise an error. Unicode characters are allowed and + will be property encoded when serialized. However, per the RFCs, unicode is + *not* allowed in the username portion of the address. + + .. attribute:: display_name + + The display name portion of the address, if any, with all quoting + removed. If the address does not have a display name, this attribute + will be an empty string. + + .. attribute:: username + + The ``username`` portion of the address, with all quoting removed. + + .. attribute:: domain + + The ``domain`` portion of the address. + + .. attribute:: addr_spec + + The ``username@domain`` portion of the address, correctly quoted + for use as a bare address (the second form shown above). This + attribute is not mutable. + + .. method:: __str__() + + The ``str`` value of the object is the address quoted according to + :rfc:`5322` rules, but with no Content Transfer Encoding of any non-ASCII + characters. + + To support SMTP (:rfc:`5321`), ``Address`` handles one special case: if + ``username`` and ``domain`` are both the empty string (or ``None``), then + the string value of the ``Address`` is ``<>``. + + +.. class:: Group(display_name=None, addresses=None) + + The class used to represent an address group. The general form of an + address group is:: + + display_name: [address-list]; + + As a convenience for processing lists of addresses that consist of a mixture + of groups and single addresses, a ``Group`` may also be used to represent + single addresses that are not part of a group by setting *display_name* to + ``None`` and providing a list of the single address as *addresses*. + + .. attribute:: display_name + + The ``display_name`` of the group. If it is ``None`` and there is + exactly one ``Address`` in ``addresses``, then the ``Group`` represents a + single address that is not in a group. + + .. attribute:: addresses + + A possibly empty tuple of :class:`.Address` objects representing the + addresses in the group. + + .. method:: __str__() + + The ``str`` value of a ``Group`` is formatted according to :rfc:`5322`, + but with no Content Transfer Encoding of any non-ASCII characters. If + ``display_name`` is none and there is a single ``Address`` in the + ``addresses`` list, the ``str`` value will be the same as the ``str`` of + that single ``Address``. + + +.. rubric:: Footnotes + +.. [1] Originally added in 3.3 as a :term:`provisional module ` diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst index 838bcf529aff20..9b1f79c1a240ab 100644 --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -3,6 +3,94 @@ .. module:: email.message :synopsis: The base class representing email messages. +<<<<<<< HEAD +======= +.. moduleauthor:: R. David Murray +.. sectionauthor:: R. David Murray , + Barry A. Warsaw + +**Source code:** :source:`Lib/email/message.py` + +-------------- + +.. versionadded:: 3.6 [1]_ + +The central class in the :mod:`email` package is the :class:`EmailMessage` +class, imported from the :mod:`email.message` module. It is the base class for +the :mod:`email` object model. :class:`EmailMessage` provides the core +functionality for setting and querying header fields, for accessing message +bodies, and for creating or modifying structured messages. + +An email message consists of *headers* and a *payload* (which is also referred +to as the *content*). Headers are :rfc:`5322` or :rfc:`6532` style field names +and values, where the field name and value are separated by a colon. The colon +is not part of either the field name or the field value. The payload may be a +simple text message, or a binary object, or a structured sequence of +sub-messages each with their own set of headers and their own payload. The +latter type of payload is indicated by the message having a MIME type such as +:mimetype:`multipart/\*` or :mimetype:`message/rfc822`. + +The conceptual model provided by an :class:`EmailMessage` object is that of an +ordered dictionary of headers coupled with a *payload* that represents the +:rfc:`5322` body of the message, which might be a list of sub-``EmailMessage`` +objects. In addition to the normal dictionary methods for accessing the header +names and values, there are methods for accessing specialized information from +the headers (for example the MIME content type), for operating on the payload, +for generating a serialized version of the message, and for recursively walking +over the object tree. + +The :class:`EmailMessage` dictionary-like interface is indexed by the header +names, which must be ASCII values. The values of the dictionary are strings +with some extra methods. Headers are stored and returned in case-preserving +form, but field names are matched case-insensitively. Unlike a real dict, +there is an ordering to the keys, and there can be duplicate keys. Additional +methods are provided for working with headers that have duplicate keys. + +The *payload* is either a string or bytes object, in the case of simple message +objects, or a list of :class:`EmailMessage` objects, for MIME container +documents such as :mimetype:`multipart/\*` and :mimetype:`message/rfc822` +message objects. + + +.. class:: EmailMessage(policy=default) + + If *policy* is specified use the rules it specifies to update and serialize + the representation of the message. If *policy* is not set, use the + :class:`~email.policy.default` policy, which follows the rules of the email + RFCs except for line endings (instead of the RFC mandated ``\r\n``, it uses + the Python standard ``\n`` line endings). For more information see the + :mod:`~email.policy` documentation. + + .. method:: as_string(unixfrom=False, maxheaderlen=None, policy=None) + + Return the entire message flattened as a string. When optional + *unixfrom* is true, the envelope header is included in the returned + string. *unixfrom* defaults to ``False``. For backward compatibility + with the base :class:`~email.message.Message` class *maxheaderlen* is + accepted, but defaults to ``None``, which means that by default the line + length is controlled by the + :attr:`~email.policy.EmailPolicy.max_line_length` of the policy. The + *policy* argument may be used to override the default policy obtained + from the message instance. This can be used to control some of the + formatting produced by the method, since the specified *policy* will be + passed to the :class:`~email.generator.Generator`. + + Flattening the message may trigger changes to the :class:`EmailMessage` + if defaults need to be filled in to complete the transformation to a + string (for example, MIME boundaries may be generated or modified). + + Note that this method is provided as a convenience and may not be the + most useful way to serialize messages in your application, especially if + you are dealing with multiple messages. See + :class:`email.generator.Generator` for a more flexible API for + serializing messages. Note also that this method is restricted to + producing messages serialized as "7 bit clean" when + :attr:`~email.policy.EmailPolicy.utf8` is ``False``, which is the default. + + .. versionchanged:: 3.6 the default behavior when *maxheaderlen* + is not specified was changed from defaulting to 0 to defaulting + to the value of *max_line_length* from the policy. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) The central class in the :mod:`email` package is the :class:`Message` class, @@ -224,6 +312,16 @@ Here are the methods of the :class:`Message` class: del msg['subject'] msg['subject'] = 'Python roolz!' +<<<<<<< HEAD +======= + If the :mod:`policy` defines certain headers to be unique (as the standard + policies do), this method may raise a :exc:`ValueError` when an attempt + is made to assign a value to such a header when one already exists. This + behavior is intentional for consistency's sake, but do not depend on it + as we may choose to make such assignments do an automatic deletion of the + existing header in the future. + +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) .. method:: __delitem__(name) @@ -549,6 +647,104 @@ Here are the methods of the :class:`Message` class: text/plain text/plain message/rfc822 +<<<<<<< HEAD +======= + text/plain + + ``walk`` iterates over the subparts of any part where + :meth:`is_multipart` returns ``True``, even though + ``msg.get_content_maintype() == 'multipart'`` may return ``False``. We + can see this in our example by making use of the ``_structure`` debug + helper function: + + .. doctest:: + + >>> for part in msg.walk(): + ... print(part.get_content_maintype() == 'multipart', + ... part.is_multipart()) + True True + False False + False True + False False + False False + False True + False False + >>> _structure(msg) + multipart/report + text/plain + message/delivery-status + text/plain + text/plain + message/rfc822 + text/plain + + Here the ``message`` parts are not ``multiparts``, but they do contain + subparts. ``is_multipart()`` returns ``True`` and ``walk`` descends + into the subparts. + + + .. method:: get_body(preferencelist=('related', 'html', 'plain')) + + Return the MIME part that is the best candidate to be the "body" of the + message. + + *preferencelist* must be a sequence of strings from the set ``related``, + ``html``, and ``plain``, and indicates the order of preference for the + content type of the part returned. + + Start looking for candidate matches with the object on which the + ``get_body`` method is called. + + If ``related`` is not included in *preferencelist*, consider the root + part (or subpart of the root part) of any related encountered as a + candidate if the (sub-)part matches a preference. + + When encountering a ``multipart/related``, check the ``start`` parameter + and if a part with a matching :mailheader:`Content-ID` is found, consider + only it when looking for candidate matches. Otherwise consider only the + first (default root) part of the ``multipart/related``. + + If a part has a :mailheader:`Content-Disposition` header, only consider + the part a candidate match if the value of the header is ``inline``. + + If none of the candidates matches any of the preferences in + *preferencelist*, return ``None``. + + Notes: (1) For most applications the only *preferencelist* combinations + that really make sense are ``('plain',)``, ``('html', 'plain')``, and the + default ``('related', 'html', 'plain')``. (2) Because matching starts + with the object on which ``get_body`` is called, calling ``get_body`` on + a ``multipart/related`` will return the object itself unless + *preferencelist* has a non-default value. (3) Messages (or message parts) + that do not specify a :mailheader:`Content-Type` or whose + :mailheader:`Content-Type` header is invalid will be treated as if they + are of type ``text/plain``, which may occasionally cause ``get_body`` to + return unexpected results. + + + .. method:: iter_attachments() + + Return an iterator over all of the immediate sub-parts of the message + that are not candidate "body" parts. That is, skip the first occurrence + of each of ``text/plain``, ``text/html``, ``multipart/related``, or + ``multipart/alternative`` (unless they are explicitly marked as + attachments via :mailheader:`Content-Disposition: attachment`), and + return all remaining parts. When applied directly to a + ``multipart/related``, return an iterator over the all the related parts + except the root part (ie: the part pointed to by the ``start`` parameter, + or the first part if there is no ``start`` parameter or the ``start`` + parameter doesn't match the :mailheader:`Content-ID` of any of the + parts). When applied directly to a ``multipart/alternative`` or a + non-``multipart``, return an empty iterator. + + + .. method:: iter_parts() + + Return an iterator over all of the immediate sub-parts of the message, + which will be empty for a non-``multipart``. (See also + :meth:`~email.message.EmailMessage.walk`.) + +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) .. versionchanged:: 2.5 The previously deprecated methods :meth:`get_type`, :meth:`get_main_type`, and @@ -601,3 +797,19 @@ Here are the methods of the :class:`Message` class: .. versionadded:: 2.4 +<<<<<<< HEAD +======= +.. class:: MIMEPart(policy=default) + + This class represents a subpart of a MIME message. It is identical to + :class:`EmailMessage`, except that no :mailheader:`MIME-Version` headers are + added when :meth:`~EmailMessage.set_content` is called, since sub-parts do + not need their own :mailheader:`MIME-Version` headers. + + +.. rubric:: Footnotes + +.. [1] Originally added in 3.4 as a :term:`provisional module `. Docs for legacy message class moved to + :ref:`compat32_message`. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index dcf7b59c05e303..5cbb4dd39eaf24 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -210,7 +210,7 @@ Here are the classes: Unless the ``_charset`` parameter is explicitly set to ``None``, the MIMEText object created will have both a :mailheader:`Content-Type` header - with a ``charset`` parameter, and a :mailheader:`Content-Transfer-Endcoding` + with a ``charset`` parameter, and a :mailheader:`Content-Transfer-Encoding` header. This means that a subsequent ``set_payload`` call will not result in an encoded payload, even if a charset is passed in the ``set_payload`` command. You can "reset" this behavior by deleting the diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst index 04e23345977b57..7ab59062d95851 100644 --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -116,8 +116,14 @@ class. effectively non-strict. You should simply stop passing a *strict* flag to the :class:`Parser` constructor. +<<<<<<< HEAD .. versionchanged:: 2.2.2 The *strict* flag was added. +======= + Create a :class:`BytesParser` instance. The *_class* and *policy* + arguments have the same meaning and semantics as the *_factory* + and *policy* arguments of :class:`BytesFeedParser`. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) .. versionchanged:: 2.4 The *strict* flag was deprecated. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst new file mode 100644 index 00000000000000..8e7076259810f5 --- /dev/null +++ b/Doc/library/email.policy.rst @@ -0,0 +1,651 @@ +:mod:`email.policy`: Policy Objects +----------------------------------- + +.. module:: email.policy + :synopsis: Controlling the parsing and generating of messages + +.. moduleauthor:: R. David Murray +.. sectionauthor:: R. David Murray + +.. versionadded:: 3.3 + +**Source code:** :source:`Lib/email/policy.py` + +-------------- + +The :mod:`email` package's prime focus is the handling of email messages as +described by the various email and MIME RFCs. However, the general format of +email messages (a block of header fields each consisting of a name followed by +a colon followed by a value, the whole block followed by a blank line and an +arbitrary 'body'), is a format that has found utility outside of the realm of +email. Some of these uses conform fairly closely to the main email RFCs, some +do not. Even when working with email, there are times when it is desirable to +break strict compliance with the RFCs, such as generating emails that +interoperate with email servers that do not themselves follow the standards, or +that implement extensions you want to use in ways that violate the +standards. + +Policy objects give the email package the flexibility to handle all these +disparate use cases. + +A :class:`Policy` object encapsulates a set of attributes and methods that +control the behavior of various components of the email package during use. +:class:`Policy` instances can be passed to various classes and methods in the +email package to alter the default behavior. The settable values and their +defaults are described below. + +There is a default policy used by all classes in the email package. For all of +the :mod:`~email.parser` classes and the related convenience functions, and for +the :class:`~email.message.Message` class, this is the :class:`Compat32` +policy, via its corresponding pre-defined instance :const:`compat32`. This +policy provides for complete backward compatibility (in some cases, including +bug compatibility) with the pre-Python3.3 version of the email package. + +This default value for the *policy* keyword to +:class:`~email.message.EmailMessage` is the :class:`EmailPolicy` policy, via +its pre-defined instance :data:`~default`. + +When a :class:`~email.message.Message` or :class:`~email.message.EmailMessage` +object is created, it acquires a policy. If the message is created by a +:mod:`~email.parser`, a policy passed to the parser will be the policy used by +the message it creates. If the message is created by the program, then the +policy can be specified when it is created. When a message is passed to a +:mod:`~email.generator`, the generator uses the policy from the message by +default, but you can also pass a specific policy to the generator that will +override the one stored on the message object. + +The default value for the *policy* keyword for the :mod:`email.parser` classes +and the parser convenience functions **will be changing** in a future version of +Python. Therefore you should **always specify explicitly which policy you want +to use** when calling any of the classes and functions described in the +:mod:`~email.parser` module. + +The first part of this documentation covers the features of :class:`Policy`, an +:term:`abstract base class` that defines the features that are common to all +policy objects, including :const:`compat32`. This includes certain hook +methods that are called internally by the email package, which a custom policy +could override to obtain different behavior. The second part describes the +concrete classes :class:`EmailPolicy` and :class:`Compat32`, which implement +the hooks that provide the standard behavior and the backward compatible +behavior and features, respectively. + +:class:`Policy` instances are immutable, but they can be cloned, accepting the +same keyword arguments as the class constructor and returning a new +:class:`Policy` instance that is a copy of the original but with the specified +attributes values changed. + +As an example, the following code could be used to read an email message from a +file on disk and pass it to the system ``sendmail`` program on a Unix system: + +.. testsetup:: + + from unittest import mock + mocker = mock.patch('subprocess.Popen') + m = mocker.start() + proc = mock.MagicMock() + m.return_value = proc + proc.stdin.close.return_value = None + mymsg = open('mymsg.txt', 'w') + mymsg.write('To: abc@xyz.com\n\n') + mymsg.flush() + +.. doctest:: + + >>> from email import message_from_binary_file + >>> from email.generator import BytesGenerator + >>> from email import policy + >>> from subprocess import Popen, PIPE + >>> with open('mymsg.txt', 'rb') as f: + ... msg = message_from_binary_file(f, policy=policy.default) + >>> p = Popen(['sendmail', msg['To'].addresses[0]], stdin=PIPE) + >>> g = BytesGenerator(p.stdin, policy=msg.policy.clone(linesep='\r\n')) + >>> g.flatten(msg) + >>> p.stdin.close() + >>> rc = p.wait() + +.. testcleanup:: + + mymsg.close() + mocker.stop() + import os + os.remove('mymsg.txt') + +Here we are telling :class:`~email.generator.BytesGenerator` to use the RFC +correct line separator characters when creating the binary string to feed into +``sendmail's`` ``stdin``, where the default policy would use ``\n`` line +separators. + +Some email package methods accept a *policy* keyword argument, allowing the +policy to be overridden for that method. For example, the following code uses +the :meth:`~email.message.Message.as_bytes` method of the *msg* object from +the previous example and writes the message to a file using the native line +separators for the platform on which it is running:: + + >>> import os + >>> with open('converted.txt', 'wb') as f: + ... f.write(msg.as_bytes(policy=msg.policy.clone(linesep=os.linesep))) + 17 + +Policy objects can also be combined using the addition operator, producing a +policy object whose settings are a combination of the non-default values of the +summed objects:: + + >>> compat_SMTP = policy.compat32.clone(linesep='\r\n') + >>> compat_strict = policy.compat32.clone(raise_on_defect=True) + >>> compat_strict_SMTP = compat_SMTP + compat_strict + +This operation is not commutative; that is, the order in which the objects are +added matters. To illustrate:: + + >>> policy100 = policy.compat32.clone(max_line_length=100) + >>> policy80 = policy.compat32.clone(max_line_length=80) + >>> apolicy = policy100 + policy80 + >>> apolicy.max_line_length + 80 + >>> apolicy = policy80 + policy100 + >>> apolicy.max_line_length + 100 + + +.. class:: Policy(**kw) + + This is the :term:`abstract base class` for all policy classes. It provides + default implementations for a couple of trivial methods, as well as the + implementation of the immutability property, the :meth:`clone` method, and + the constructor semantics. + + The constructor of a policy class can be passed various keyword arguments. + The arguments that may be specified are any non-method properties on this + class, plus any additional non-method properties on the concrete class. A + value specified in the constructor will override the default value for the + corresponding attribute. + + This class defines the following properties, and thus values for the + following may be passed in the constructor of any policy class: + + + .. attribute:: max_line_length + + The maximum length of any line in the serialized output, not counting the + end of line character(s). Default is 78, per :rfc:`5322`. A value of + ``0`` or :const:`None` indicates that no line wrapping should be + done at all. + + + .. attribute:: linesep + + The string to be used to terminate lines in serialized output. The + default is ``\n`` because that's the internal end-of-line discipline used + by Python, though ``\r\n`` is required by the RFCs. + + + .. attribute:: cte_type + + Controls the type of Content Transfer Encodings that may be or are + required to be used. The possible values are: + + .. tabularcolumns:: |l|L| + + ======== =============================================================== + ``7bit`` all data must be "7 bit clean" (ASCII-only). This means that + where necessary data will be encoded using either + quoted-printable or base64 encoding. + + ``8bit`` data is not constrained to be 7 bit clean. Data in headers is + still required to be ASCII-only and so will be encoded (see + :meth:`fold_binary` and :attr:`~EmailPolicy.utf8` below for + exceptions), but body parts may use the ``8bit`` CTE. + ======== =============================================================== + + A ``cte_type`` value of ``8bit`` only works with ``BytesGenerator``, not + ``Generator``, because strings cannot contain binary data. If a + ``Generator`` is operating under a policy that specifies + ``cte_type=8bit``, it will act as if ``cte_type`` is ``7bit``. + + + .. attribute:: raise_on_defect + + If :const:`True`, any defects encountered will be raised as errors. If + :const:`False` (the default), defects will be passed to the + :meth:`register_defect` method. + + + .. attribute:: mangle_from\_ + + If :const:`True`, lines starting with *"From "* in the body are + escaped by putting a ``>`` in front of them. This parameter is used when + the message is being serialized by a generator. + Default: :const:`False`. + + .. versionadded:: 3.5 + The *mangle_from_* parameter. + + + .. attribute:: message_factory + + A factory function for constructing a new empty message object. Used + by the parser when building messages. Defaults to ``None``, in + which case :class:`~email.message.Message` is used. + + .. versionadded:: 3.6 + + The following :class:`Policy` method is intended to be called by code using + the email library to create policy instances with custom settings: + + + .. method:: clone(**kw) + + Return a new :class:`Policy` instance whose attributes have the same + values as the current instance, except where those attributes are + given new values by the keyword arguments. + + + The remaining :class:`Policy` methods are called by the email package code, + and are not intended to be called by an application using the email package. + A custom policy must implement all of these methods. + + + .. method:: handle_defect(obj, defect) + + Handle a *defect* found on *obj*. When the email package calls this + method, *defect* will always be a subclass of + :class:`~email.errors.Defect`. + + The default implementation checks the :attr:`raise_on_defect` flag. If + it is ``True``, *defect* is raised as an exception. If it is ``False`` + (the default), *obj* and *defect* are passed to :meth:`register_defect`. + + + .. method:: register_defect(obj, defect) + + Register a *defect* on *obj*. In the email package, *defect* will always + be a subclass of :class:`~email.errors.Defect`. + + The default implementation calls the ``append`` method of the ``defects`` + attribute of *obj*. When the email package calls :attr:`handle_defect`, + *obj* will normally have a ``defects`` attribute that has an ``append`` + method. Custom object types used with the email package (for example, + custom ``Message`` objects) should also provide such an attribute, + otherwise defects in parsed messages will raise unexpected errors. + + + .. method:: header_max_count(name) + + Return the maximum allowed number of headers named *name*. + + Called when a header is added to an :class:`~email.message.EmailMessage` + or :class:`~email.message.Message` object. If the returned value is not + ``0`` or ``None``, and there are already a number of headers with the + name *name* greater than or equal to the value returned, a + :exc:`ValueError` is raised. + + Because the default behavior of ``Message.__setitem__`` is to append the + value to the list of headers, it is easy to create duplicate headers + without realizing it. This method allows certain headers to be limited + in the number of instances of that header that may be added to a + ``Message`` programmatically. (The limit is not observed by the parser, + which will faithfully produce as many headers as exist in the message + being parsed.) + + The default implementation returns ``None`` for all header names. + + + .. method:: header_source_parse(sourcelines) + + The email package calls this method with a list of strings, each string + ending with the line separation characters found in the source being + parsed. The first line includes the field header name and separator. + All whitespace in the source is preserved. The method should return the + ``(name, value)`` tuple that is to be stored in the ``Message`` to + represent the parsed header. + + If an implementation wishes to retain compatibility with the existing + email package policies, *name* should be the case preserved name (all + characters up to the '``:``' separator), while *value* should be the + unfolded value (all line separator characters removed, but whitespace + kept intact), stripped of leading whitespace. + + *sourcelines* may contain surrogateescaped binary data. + + There is no default implementation + + + .. method:: header_store_parse(name, value) + + The email package calls this method with the name and value provided by + the application program when the application program is modifying a + ``Message`` programmatically (as opposed to a ``Message`` created by a + parser). The method should return the ``(name, value)`` tuple that is to + be stored in the ``Message`` to represent the header. + + If an implementation wishes to retain compatibility with the existing + email package policies, the *name* and *value* should be strings or + string subclasses that do not change the content of the passed in + arguments. + + There is no default implementation + + + .. method:: header_fetch_parse(name, value) + + The email package calls this method with the *name* and *value* currently + stored in the ``Message`` when that header is requested by the + application program, and whatever the method returns is what is passed + back to the application as the value of the header being retrieved. + Note that there may be more than one header with the same name stored in + the ``Message``; the method is passed the specific name and value of the + header destined to be returned to the application. + + *value* may contain surrogateescaped binary data. There should be no + surrogateescaped binary data in the value returned by the method. + + There is no default implementation + + + .. method:: fold(name, value) + + The email package calls this method with the *name* and *value* currently + stored in the ``Message`` for a given header. The method should return a + string that represents that header "folded" correctly (according to the + policy settings) by composing the *name* with the *value* and inserting + :attr:`linesep` characters at the appropriate places. See :rfc:`5322` + for a discussion of the rules for folding email headers. + + *value* may contain surrogateescaped binary data. There should be no + surrogateescaped binary data in the string returned by the method. + + + .. method:: fold_binary(name, value) + + The same as :meth:`fold`, except that the returned value should be a + bytes object rather than a string. + + *value* may contain surrogateescaped binary data. These could be + converted back into binary data in the returned bytes object. + + + +.. class:: EmailPolicy(**kw) + + This concrete :class:`Policy` provides behavior that is intended to be fully + compliant with the current email RFCs. These include (but are not limited + to) :rfc:`5322`, :rfc:`2047`, and the current MIME RFCs. + + This policy adds new header parsing and folding algorithms. Instead of + simple strings, headers are ``str`` subclasses with attributes that depend + on the type of the field. The parsing and folding algorithm fully implement + :rfc:`2047` and :rfc:`5322`. + + The default value for the :attr:`~email.policy.Policy.message_factory` + attribute is :class:`~email.message.EmailMessage`. + + In addition to the settable attributes listed above that apply to all + policies, this policy adds the following additional attributes: + + .. versionadded:: 3.6 [1]_ + + + .. attribute:: utf8 + + If ``False``, follow :rfc:`5322`, supporting non-ASCII characters in + headers by encoding them as "encoded words". If ``True``, follow + :rfc:`6532` and use ``utf-8`` encoding for headers. Messages + formatted in this way may be passed to SMTP servers that support + the ``SMTPUTF8`` extension (:rfc:`6531`). + + + .. attribute:: refold_source + + If the value for a header in the ``Message`` object originated from a + :mod:`~email.parser` (as opposed to being set by a program), this + attribute indicates whether or not a generator should refold that value + when transforming the message back into serialized form. The possible + values are: + + ======== =============================================================== + ``none`` all source values use original folding + + ``long`` source values that have any line that is longer than + ``max_line_length`` will be refolded + + ``all`` all values are refolded. + ======== =============================================================== + + The default is ``long``. + + + .. attribute:: header_factory + + A callable that takes two arguments, ``name`` and ``value``, where + ``name`` is a header field name and ``value`` is an unfolded header field + value, and returns a string subclass that represents that header. A + default ``header_factory`` (see :mod:`~email.headerregistry`) is provided + that supports custom parsing for the various address and date :RFC:`5322` + header field types, and the major MIME header field stypes. Support for + additional custom parsing will be added in the future. + + + .. attribute:: content_manager + + An object with at least two methods: get_content and set_content. When + the :meth:`~email.message.EmailMessage.get_content` or + :meth:`~email.message.EmailMessage.set_content` method of an + :class:`~email.message.EmailMessage` object is called, it calls the + corresponding method of this object, passing it the message object as its + first argument, and any arguments or keywords that were passed to it as + additional arguments. By default ``content_manager`` is set to + :data:`~email.contentmanager.raw_data_manager`. + + .. versionadded:: 3.4 + + + The class provides the following concrete implementations of the abstract + methods of :class:`Policy`: + + + .. method:: header_max_count(name) + + Returns the value of the + :attr:`~email.headerregistry.BaseHeader.max_count` attribute of the + specialized class used to represent the header with the given name. + + + .. method:: header_source_parse(sourcelines) + + + The name is parsed as everything up to the '``:``' and returned + unmodified. The value is determined by stripping leading whitespace off + the remainder of the first line, joining all subsequent lines together, + and stripping any trailing carriage return or linefeed characters. + + + .. method:: header_store_parse(name, value) + + The name is returned unchanged. If the input value has a ``name`` + attribute and it matches *name* ignoring case, the value is returned + unchanged. Otherwise the *name* and *value* are passed to + ``header_factory``, and the resulting header object is returned as + the value. In this case a ``ValueError`` is raised if the input value + contains CR or LF characters. + + + .. method:: header_fetch_parse(name, value) + + If the value has a ``name`` attribute, it is returned to unmodified. + Otherwise the *name*, and the *value* with any CR or LF characters + removed, are passed to the ``header_factory``, and the resulting + header object is returned. Any surrogateescaped bytes get turned into + the unicode unknown-character glyph. + + + .. method:: fold(name, value) + + Header folding is controlled by the :attr:`refold_source` policy setting. + A value is considered to be a 'source value' if and only if it does not + have a ``name`` attribute (having a ``name`` attribute means it is a + header object of some sort). If a source value needs to be refolded + according to the policy, it is converted into a header object by + passing the *name* and the *value* with any CR and LF characters removed + to the ``header_factory``. Folding of a header object is done by + calling its ``fold`` method with the current policy. + + Source values are split into lines using :meth:`~str.splitlines`. If + the value is not to be refolded, the lines are rejoined using the + ``linesep`` from the policy and returned. The exception is lines + containing non-ascii binary data. In that case the value is refolded + regardless of the ``refold_source`` setting, which causes the binary data + to be CTE encoded using the ``unknown-8bit`` charset. + + + .. method:: fold_binary(name, value) + + The same as :meth:`fold` if :attr:`~Policy.cte_type` is ``7bit``, except + that the returned value is bytes. + + If :attr:`~Policy.cte_type` is ``8bit``, non-ASCII binary data is + converted back + into bytes. Headers with binary data are not refolded, regardless of the + ``refold_header`` setting, since there is no way to know whether the + binary data consists of single byte characters or multibyte characters. + + +The following instances of :class:`EmailPolicy` provide defaults suitable for +specific application domains. Note that in the future the behavior of these +instances (in particular the ``HTTP`` instance) may be adjusted to conform even +more closely to the RFCs relevant to their domains. + + +.. data:: default + + An instance of ``EmailPolicy`` with all defaults unchanged. This policy + uses the standard Python ``\n`` line endings rather than the RFC-correct + ``\r\n``. + + +.. data:: SMTP + + Suitable for serializing messages in conformance with the email RFCs. + Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC + compliant. + + +.. data:: SMTPUTF8 + + The same as ``SMTP`` except that :attr:`~EmailPolicy.utf8` is ``True``. + Useful for serializing messages to a message store without using encoded + words in the headers. Should only be used for SMTP transmission if the + sender or recipient addresses have non-ASCII characters (the + :meth:`smtplib.SMTP.send_message` method handles this automatically). + + +.. data:: HTTP + + Suitable for serializing headers with for use in HTTP traffic. Like + ``SMTP`` except that ``max_line_length`` is set to ``None`` (unlimited). + + +.. data:: strict + + Convenience instance. The same as ``default`` except that + ``raise_on_defect`` is set to ``True``. This allows any policy to be made + strict by writing:: + + somepolicy + policy.strict + + +With all of these :class:`EmailPolicies <.EmailPolicy>`, the effective API of +the email package is changed from the Python 3.2 API in the following ways: + + * Setting a header on a :class:`~email.message.Message` results in that + header being parsed and a header object created. + + * Fetching a header value from a :class:`~email.message.Message` results + in that header being parsed and a header object created and + returned. + + * Any header object, or any header that is refolded due to the + policy settings, is folded using an algorithm that fully implements the + RFC folding algorithms, including knowing where encoded words are required + and allowed. + +From the application view, this means that any header obtained through the +:class:`~email.message.EmailMessage` is a header object with extra +attributes, whose string value is the fully decoded unicode value of the +header. Likewise, a header may be assigned a new value, or a new header +created, using a unicode string, and the policy will take care of converting +the unicode string into the correct RFC encoded form. + +The header objects and their attributes are described in +:mod:`~email.headerregistry`. + + + +.. class:: Compat32(**kw) + + This concrete :class:`Policy` is the backward compatibility policy. It + replicates the behavior of the email package in Python 3.2. The + :mod:`~email.policy` module also defines an instance of this class, + :const:`compat32`, that is used as the default policy. Thus the default + behavior of the email package is to maintain compatibility with Python 3.2. + + The following attributes have values that are different from the + :class:`Policy` default: + + + .. attribute:: mangle_from_ + + The default is ``True``. + + + The class provides the following concrete implementations of the + abstract methods of :class:`Policy`: + + + .. method:: header_source_parse(sourcelines) + + The name is parsed as everything up to the '``:``' and returned + unmodified. The value is determined by stripping leading whitespace off + the remainder of the first line, joining all subsequent lines together, + and stripping any trailing carriage return or linefeed characters. + + + .. method:: header_store_parse(name, value) + + The name and value are returned unmodified. + + + .. method:: header_fetch_parse(name, value) + + If the value contains binary data, it is converted into a + :class:`~email.header.Header` object using the ``unknown-8bit`` charset. + Otherwise it is returned unmodified. + + + .. method:: fold(name, value) + + Headers are folded using the :class:`~email.header.Header` folding + algorithm, which preserves existing line breaks in the value, and wraps + each resulting line to the ``max_line_length``. Non-ASCII binary data are + CTE encoded using the ``unknown-8bit`` charset. + + + .. method:: fold_binary(name, value) + + Headers are folded using the :class:`~email.header.Header` folding + algorithm, which preserves existing line breaks in the value, and wraps + each resulting line to the ``max_line_length``. If ``cte_type`` is + ``7bit``, non-ascii binary data is CTE encoded using the ``unknown-8bit`` + charset. Otherwise the original source header is used, with its existing + line breaks and any (RFC invalid) binary data it may contain. + + +.. data:: compat32 + + An instance of :class:`Compat32`, providing backward compatibility with the + behavior of the email package in Python 3.2. + + +.. rubric:: Footnotes + +.. [1] Originally added in 3.3 as a :term:`provisional feature `. diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index fe14656008177f..f5fe36db130beb 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -241,6 +241,12 @@ The following exceptions are the exceptions that are actually raised. .. versionadded:: 1.5.2 +<<<<<<< HEAD +======= + It should not be used to indicate that an operator or method is not + meant to be supported at all -- in that case either leave the operator / + method undefined or, if a subclass, set it to :data:`None`. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) .. exception:: OSError diff --git a/Doc/library/sunau.rst b/Doc/library/sunau.rst index 5d57d4b95784f9..5bf48033a73ccb 100644 --- a/Doc/library/sunau.rst +++ b/Doc/library/sunau.rst @@ -117,7 +117,7 @@ AU_read objects, as returned by :func:`.open` above, have the following methods: .. method:: AU_read.getnchannels() - Returns number of audio channels (1 for mone, 2 for stereo). + Returns number of audio channels (1 for mono, 2 for stereo). .. method:: AU_read.getsampwidth() diff --git a/Doc/library/turtle.rst b/Doc/library/turtle.rst index ca4f37e478f2ef..8d2600fd364afd 100644 --- a/Doc/library/turtle.rst +++ b/Doc/library/turtle.rst @@ -1697,6 +1697,56 @@ Using screen events >>> running = False +<<<<<<< HEAD +======= +.. function:: mainloop() + done() + + Starts event loop - calling Tkinter's mainloop function. + Must be the last statement in a turtle graphics program. + Must *not* be used if a script is run from within IDLE in -n mode + (No subprocess) - for interactive use of turtle graphics. :: + + >>> screen.mainloop() + + +Input methods +------------- + +.. function:: textinput(title, prompt) + + :param title: string + :param prompt: string + + Pop up a dialog window for input of a string. Parameter title is + the title of the dialog window, prompt is a text mostly describing + what information to input. + Return the string input. If the dialog is canceled, return ``None``. :: + + >>> screen.textinput("NIM", "Name of first player:") + + +.. function:: numinput(title, prompt, default=None, minval=None, maxval=None) + + :param title: string + :param prompt: string + :param default: number (optional) + :param minval: number (optional) + :param maxval: number (optional) + + Pop up a dialog window for input of a number. title is the title of the + dialog window, prompt is a text mostly describing what numerical information + to input. default: default value, minval: minimum value for input, + maxval: maximum value for input + The number input must be in the range minval .. maxval if these are + given. If not, a hint is issued and the dialog remains open for + correction. + Return the number input. If the dialog is canceled, return ``None``. :: + + >>> screen.numinput("Poker", "Your stakes:", 1000, minval=10, maxval=10000) + + +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) Settings and special methods ---------------------------- diff --git a/Doc/library/xml.dom.pulldom.rst b/Doc/library/xml.dom.pulldom.rst index 9032706fcd8db8..78e5e1238fdf6c 100644 --- a/Doc/library/xml.dom.pulldom.rst +++ b/Doc/library/xml.dom.pulldom.rst @@ -67,6 +67,16 @@ DOMEventStream Objects ... +<<<<<<< HEAD +======= + Return a tuple containing *event* and the current *node* as + :class:`xml.dom.minidom.Document` if event equals :data:`START_DOCUMENT`, + :class:`xml.dom.minidom.Element` if event equals :data:`START_ELEMENT` or + :data:`END_ELEMENT` or :class:`xml.dom.minidom.Text` if event equals + :data:`CHARACTERS`. + The current node does not contain information about its children, unless + :func:`expandNode` is called. +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) .. method:: DOMEventStream.expandNode(node) diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 8556fa810f6cb3..087e580f51d35e 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -505,6 +505,124 @@ generator functions:: >>> generator.close() Don't forget to clean up when 'close()' is called. +<<<<<<< HEAD +======= +For examples using ``yield from``, see :ref:`pep-380` in "What's New in +Python." + +.. _asynchronous-generator-functions: + +Asynchronous generator functions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The presence of a yield expression in a function or method defined using +:keyword:`async def` further defines the function as a +:term:`asynchronous generator` function. + +When an asynchronous generator function is called, it returns an +asynchronous iterator known as an asynchronous generator object. +That object then controls the execution of the generator function. +An asynchronous generator object is typically used in an +:keyword:`async for` statement in a coroutine function analogously to +how a generator object would be used in a :keyword:`for` statement. + +Calling one of the asynchronous generator's methods returns an +:term:`awaitable` object, and the execution starts when this object +is awaited on. At that time, the execution proceeds to the first yield +expression, where it is suspended again, returning the value of +:token:`expression_list` to the awaiting coroutine. As with a generator, +suspension means that all local state is retained, including the +current bindings of local variables, the instruction pointer, the internal +evaluation stack, and the state of any exception handling. When the execution +is resumed by awaiting on the next object returned by the asynchronous +generator's methods, the function can proceed exactly as if the yield +expression were just another external call. The value of the yield expression +after resuming depends on the method which resumed the execution. If +:meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if +:meth:`~agen.asend` is used, then the result will be the value passed in to +that method. + +In an asynchronous generator function, yield expressions are allowed anywhere +in a :keyword:`try` construct. However, if an asynchronous generator is not +resumed before it is finalized (by reaching a zero reference count or by +being garbage collected), then a yield expression within a :keyword:`try` +construct could result in a failure to execute pending :keyword:`finally` +clauses. In this case, it is the responsibility of the event loop or +scheduler running the asynchronous generator to call the asynchronous +generator-iterator's :meth:`~agen.aclose` method and run the resulting +coroutine object, thus allowing any pending :keyword:`finally` clauses +to execute. + +To take care of finalization, an event loop should define +a *finalizer* function which takes an asynchronous generator-iterator +and presumably calls :meth:`~agen.aclose` and executes the coroutine. +This *finalizer* may be registered by calling :func:`sys.set_asyncgen_hooks`. +When first iterated over, an asynchronous generator-iterator will store the +registered *finalizer* to be called upon finalization. For a reference example +of a *finalizer* method see the implementation of +``asyncio.Loop.shutdown_asyncgens`` in :source:`Lib/asyncio/base_events.py`. + +The expression ``yield from `` is a syntax error when used in an +asynchronous generator function. + +.. index:: object: asynchronous-generator +.. _asynchronous-generator-methods: + +Asynchronous generator-iterator methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This subsection describes the methods of an asynchronous generator iterator, +which are used to control the execution of a generator function. + + +.. index:: exception: StopAsyncIteration + +.. coroutinemethod:: agen.__anext__() + + Returns an awaitable which when run starts to execute the asynchronous + generator or resumes it at the last executed yield expression. When an + asynchronous generator function is resumed with a :meth:`~agen.__anext__` + method, the current yield expression always evaluates to :const:`None` in + the returned awaitable, which when run will continue to the next yield + expression. The value of the :token:`expression_list` of the yield + expression is the value of the :exc:`StopIteration` exception raised by + the completing coroutine. If the asynchronous generator exits without + yielding another value, the awaitable instead raises an + :exc:`StopAsyncIteration` exception, signalling that the asynchronous + iteration has completed. + + This method is normally called implicitly by a :keyword:`async for` loop. + + +.. coroutinemethod:: agen.asend(value) + + Returns an awaitable which when run resumes the execution of the + asynchronous generator. As with the :meth:`~generator.send()` method for a + generator, this "sends" a value into the asynchronous generator function, + and the *value* argument becomes the result of the current yield expression. + The awaitable returned by the :meth:`asend` method will return the next + value yielded by the generator as the value of the raised + :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the + asynchronous generator exits without yielding another value. When + :meth:`asend` is called to start the asynchronous + generator, it must be called with :const:`None` as the argument, + because there is no yield expression that could receive the value. + + +.. coroutinemethod:: agen.athrow(type[, value[, traceback]]) + + Returns an awaitable that raises an exception of type ``type`` at the point + where the asynchronous generator was paused, and returns the next value + yielded by the generator function as the value of the raised + :exc:`StopIteration` exception. If the asynchronous generator exits + without yielding another value, an :exc:`StopAsyncIteration` exception is + raised by the awaitable. + If the generator function does not catch the passed-in exception, or + raises a different exception, then when the awaitable is run that exception + propagates to the caller of the awaitable. + +.. index:: exception: GeneratorExit +>>>>>>> 3378b20... Fix typos in multiple `.rst` files (#1668) .. seealso:: diff --git a/Lib/email/architecture.rst b/Lib/email/architecture.rst new file mode 100644 index 00000000000000..fcd10bde1325bb --- /dev/null +++ b/Lib/email/architecture.rst @@ -0,0 +1,216 @@ +:mod:`email` Package Architecture +================================= + +Overview +-------- + +The email package consists of three major components: + + Model + An object structure that represents an email message, and provides an + API for creating, querying, and modifying a message. + + Parser + Takes a sequence of characters or bytes and produces a model of the + email message represented by those characters or bytes. + + Generator + Takes a model and turns it into a sequence of characters or bytes. The + sequence can either be intended for human consumption (a printable + unicode string) or bytes suitable for transmission over the wire. In + the latter case all data is properly encoded using the content transfer + encodings specified by the relevant RFCs. + +Conceptually the package is organized around the model. The model provides both +"external" APIs intended for use by application programs using the library, +and "internal" APIs intended for use by the Parser and Generator components. +This division is intentionally a bit fuzzy; the API described by this +documentation is all a public, stable API. This allows for an application +with special needs to implement its own parser and/or generator. + +In addition to the three major functional components, there is a third key +component to the architecture: + + Policy + An object that specifies various behavioral settings and carries + implementations of various behavior-controlling methods. + +The Policy framework provides a simple and convenient way to control the +behavior of the library, making it possible for the library to be used in a +very flexible fashion while leveraging the common code required to parse, +represent, and generate message-like objects. For example, in addition to the +default :rfc:`5322` email message policy, we also have a policy that manages +HTTP headers in a fashion compliant with :rfc:`2616`. Individual policy +controls, such as the maximum line length produced by the generator, can also +be controlled individually to meet specialized application requirements. + + +The Model +--------- + +The message model is implemented by the :class:`~email.message.Message` class. +The model divides a message into the two fundamental parts discussed by the +RFC: the header section and the body. The `Message` object acts as a +pseudo-dictionary of named headers. Its dictionary interface provides +convenient access to individual headers by name. However, all headers are kept +internally in an ordered list, so that the information about the order of the +headers in the original message is preserved. + +The `Message` object also has a `payload` that holds the body. A `payload` can +be one of two things: data, or a list of `Message` objects. The latter is used +to represent a multipart MIME message. Lists can be nested arbitrarily deeply +in order to represent the message, with all terminal leaves having non-list +data payloads. + + +Message Lifecycle +----------------- + +The general lifecycle of a message is: + + Creation + A `Message` object can be created by a Parser, or it can be + instantiated as an empty message by an application. + + Manipulation + The application may examine one or more headers, and/or the + payload, and it may modify one or more headers and/or + the payload. This may be done on the top level `Message` + object, or on any sub-object. + + Finalization + The Model is converted into a unicode or binary stream, + or the model is discarded. + + + +Header Policy Control During Lifecycle +-------------------------------------- + +One of the major controls exerted by the Policy is the management of headers +during the `Message` lifecycle. Most applications don't need to be aware of +this. + +A header enters the model in one of two ways: via a Parser, or by being set to +a specific value by an application program after the Model already exists. +Similarly, a header exits the model in one of two ways: by being serialized by +a Generator, or by being retrieved from a Model by an application program. The +Policy object provides hooks for all four of these pathways. + +The model storage for headers is a list of (name, value) tuples. + +The Parser identifies headers during parsing, and passes them to the +:meth:`~email.policy.Policy.header_source_parse` method of the Policy. The +result of that method is the (name, value) tuple to be stored in the model. + +When an application program supplies a header value (for example, through the +`Message` object `__setitem__` interface), the name and the value are passed to +the :meth:`~email.policy.Policy.header_store_parse` method of the Policy, which +returns the (name, value) tuple to be stored in the model. + +When an application program retrieves a header (through any of the dict or list +interfaces of `Message`), the name and value are passed to the +:meth:`~email.policy.Policy.header_fetch_parse` method of the Policy to +obtain the value returned to the application. + +When a Generator requests a header during serialization, the name and value are +passed to the :meth:`~email.policy.Policy.fold` method of the Policy, which +returns a string containing line breaks in the appropriate places. The +:meth:`~email.policy.Policy.cte_type` Policy control determines whether or +not Content Transfer Encoding is performed on the data in the header. There is +also a :meth:`~email.policy.Policy.binary_fold` method for use by generators +that produce binary output, which returns the folded header as binary data, +possibly folded at different places than the corresponding string would be. + + +Handling Binary Data +-------------------- + +In an ideal world all message data would conform to the RFCs, meaning that the +parser could decode the message into the idealized unicode message that the +sender originally wrote. In the real world, the email package must also be +able to deal with badly formatted messages, including messages containing +non-ASCII characters that either have no indicated character set or are not +valid characters in the indicated character set. + +Since email messages are *primarily* text data, and operations on message data +are primarily text operations (except for binary payloads of course), the model +stores all text data as unicode strings. Un-decodable binary inside text +data is handled by using the `surrogateescape` error handler of the ASCII +codec. As with the binary filenames the error handler was introduced to +handle, this allows the email package to "carry" the binary data received +during parsing along until the output stage, at which time it is regenerated +in its original form. + +This carried binary data is almost entirely an implementation detail. The one +place where it is visible in the API is in the "internal" API. A Parser must +do the `surrogateescape` encoding of binary input data, and pass that data to +the appropriate Policy method. The "internal" interface used by the Generator +to access header values preserves the `surrogateescaped` bytes. All other +interfaces convert the binary data either back into bytes or into a safe form +(losing information in some cases). + + +Backward Compatibility +---------------------- + +The :class:`~email.policy.Policy.Compat32` Policy provides backward +compatibility with version 5.1 of the email package. It does this via the +following implementation of the four+1 Policy methods described above: + +header_source_parse + Splits the first line on the colon to obtain the name, discards any spaces + after the colon, and joins the remainder of the line with all of the + remaining lines, preserving the linesep characters to obtain the value. + Trailing carriage return and/or linefeed characters are stripped from the + resulting value string. + +header_store_parse + Returns the name and value exactly as received from the application. + +header_fetch_parse + If the value contains any `surrogateescaped` binary data, return the value + as a :class:`~email.header.Header` object, using the character set + `unknown-8bit`. Otherwise just returns the value. + +fold + Uses :class:`~email.header.Header`'s folding to fold headers in the + same way the email5.1 generator did. + +binary_fold + Same as fold, but encodes to 'ascii'. + + +New Algorithm +------------- + +header_source_parse + Same as legacy behavior. + +header_store_parse + Same as legacy behavior. + +header_fetch_parse + If the value is already a header object, returns it. Otherwise, parses the + value using the new parser, and returns the resulting object as the value. + `surrogateescaped` bytes get turned into unicode unknown character code + points. + +fold + Uses the new header folding algorithm, respecting the policy settings. + surrogateescaped bytes are encoded using the ``unknown-8bit`` charset for + ``cte_type=7bit`` or ``8bit``. Returns a string. + + At some point there will also be a ``cte_type=unicode``, and for that + policy fold will serialize the idealized unicode message with RFC-like + folding, converting any surrogateescaped bytes into the unicode + unknown character glyph. + +binary_fold + Uses the new header folding algorithm, respecting the policy settings. + surrogateescaped bytes are encoded using the `unknown-8bit` charset for + ``cte_type=7bit``, and get turned back into bytes for ``cte_type=8bit``. + Returns bytes. + + At some point there will also be a ``cte_type=unicode``, and for that + policy binary_fold will serialize the message according to :rfc:``5335``. From e88f516d40a81dda9b7934440c16ae8454b690c2 Mon Sep 17 00:00:00 2001 From: Delirious Lettuce Date: Sun, 21 May 2017 15:00:42 -0600 Subject: [PATCH 2/2] [2.7] Fix typos in multiple `.rst` files (GH-1668). (cherry picked from commit 3378b2062c7fbb77a9b5e6d315d6b94160fac69a)