PEP 453: Explicit Bootstrapping of PIP in Python Installations

PEP 446: Newly Created File Descriptors Are Non-Inheritable

PEP 446 makes newly created file descriptors non-inheritable. In general, this is the behavior an application will want: when launching a new process, having currently open files also open in the new process can lead to all sorts of hard to find bugs, and potentially to security issues.

However, there are occasions when inheritance is desired. To support these cases, the following new functions and methods are available:

• os.get_inheritable(), os.set_inheritable()

• os.get_handle_inheritable(), os.set_handle_inheritable()

• socket.socket.get_inheritable(), socket.socket.set_inheritable()

Improvements to Codec Handling

Since it was first introduced, the codecs module has always been intended to operate as a type-neutral dynamic encoding and decoding system. However, its close coupling with the Python text model, especially the type restricted convenience methods on the builtin str, bytes and bytearray types, has historically obscured that fact.

As a key step in clarifying the situation, the codecs.encode() and codecs.decode() convenience functions are now properly documented in Python 2.7, 3.3 and 3.4. These functions have existed in the codecs module (and have been covered by the regression test suite) since Python 2.4, but were previously only discoverable through runtime introspection.

Unlike the convenience methods on str, bytes and bytearray, the codecs convenience functions support arbitrary codecs in both Python 2 and Python 3, rather than being limited to Unicode text encodings (in Python 3) or basestring <-> basestring conversions (in Python 2).

In Python 3.4, the interpreter is able to identify the known non-text encodings provided in the standard library and direct users towards these general purpose convenience functions when appropriate:

>>> b"abcdef".decode("hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs

>>> "hello".encode("rot13")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs

>>> open("foo.txt", encoding="hex")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs

In a related change, whenever it is feasible without breaking backwards compatibility, exceptions raised during encoding and decoding operations are wrapped in a chained exception of the same type that mentions the name of the codec responsible for producing the error:

>>> import codecs

>>> codecs.decode(b"abcdefgh", "hex")
Traceback (most recent call last):
  File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode
    return (binascii.a2b_hex(input), len(input))
binascii.Error: Non-hexadecimal digit found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found)

>>> codecs.encode("hello", "bz2")
Traceback (most recent call last):
  File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode
    return (bz2.compress(input), len(input))
  File "/usr/lib/python3.4/bz2.py", line 498, in compress
    return comp.compress(data) + comp.flush()
TypeError: 'str' does not support the buffer interface

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)

Finally, as the examples above show, these improvements have permitted the restoration of the convenience aliases for the non-Unicode codecs that were themselves restored in Python 3.2. This means that encoding binary data to and from its hexadecimal representation (for example) can now be written as:

>>> from codecs import encode, decode
>>> encode(b"hello", "hex")
b'68656c6c6f'
>>> decode(b"68656c6c6f", "hex")
b'hello'

The binary and text transforms provided in the standard library are detailed in Binary Transforms and Text Transforms.

(Contributed by Nick Coghlan in bpo-7475 , bpo-17827 , bpo-17828 and bpo-19619 .)

PEP 451: A ModuleSpec Type for the Import System

PEP 451 provides an encapsulation of the information about a module that the import machinery will use to load it (that is, a module specification). This helps simplify both the import implementation and several import-related APIs. The change is also a stepping stone for several future import-related improvements .

The public-facing changes from the PEP are entirely backward-compatible. Furthermore, they should be transparent to everyone but importer authors. Key finder and loader methods have been deprecated, but they will continue working. New importers should use the new methods described in the PEP. Existing importers should be updated to implement the new methods. See the Deprecated section for a list of methods that should be replaced and their replacements.

Other Language Changes

Some smaller changes made to the core Python language are:

• Unicode database updated to UCD version 6.3.

• min() and max() now accept a default keyword-only argument that can be used to specify the value they return if the iterable they are evaluating has no elements. (Contributed by Julian Berman in bpo-18111 .)

• Module objects are now weakref’able.

• Module __file__ attributes (and related values) should now always contain absolute paths by default, with the sole exception of __main__.__file__ when a script has been executed directly using a relative path. (Contributed by Brett Cannon in bpo-18416 .)

• All the UTF-* codecs (except UTF-7) now reject surrogates during both encoding and decoding unless the surrogatepass error handler is used, with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) and the UTF-16 encoder (which produces them while encoding non-BMP characters). (Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in bpo-12892 .)

• New German EBCDIC codec cp273. (Contributed by Michael Bierenfeld and Andrew Kuchling in bpo-1097797 .)

• New Ukrainian codec cp1125. (Contributed by Serhiy Storchaka in bpo-19668 .)

• bytes.join() and bytearray.join() now accept arbitrary buffer objects as arguments. (Contributed by Antoine Pitrou in bpo-15958 .)

• The int constructor now accepts any object that has an __index__ method for its base argument. (Contributed by Mark Dickinson in bpo-16772 .)

• Frame objects now have a clear() method that clears all references to local variables from the frame. (Contributed by Antoine Pitrou in bpo-17934 .)

• memoryview is now registered as a Sequence, and supports the reversed() builtin. (Contributed by Nick Coghlan and Claudiu Popa in bpo-18690 and bpo-19078 .)

• Signatures reported by help() have been modified and improved in several cases as a result of the introduction of Argument Clinic and other changes to the inspect and pydoc modules.

• __length_hint__() is now part of the formal language specification (see PEP 424 ). (Contributed by Armin Ronacher in bpo-16148 .)

asyncio

The new asyncio module (defined in PEP 3156 ) provides a standard pluggable event loop model for Python, providing solid asynchronous IO support in the standard library, and making it easier for other event loop implementations to interoperate with the standard library and each other.

For Python 3.4, this module is considered a provisional API.

ensurepip

The new ensurepip module is the primary infrastructure for the PEP 453 implementation. In the normal course of events end users will not need to interact with this module, but it can be used to manually bootstrap pip if the automated bootstrapping into an installation or virtual environment was declined.

ensurepip includes a bundled copy of pip, up-to-date as of the first release candidate of the release of CPython with which it ships (this applies to both maintenance releases and feature releases). ensurepip does not access the internet. If the installation has Internet access, after ensurepip is run the bundled pip can be used to upgrade pip to a more recent release than the bundled one. (Note that such an upgraded version of pip is considered to be a separately installed package and will not be removed if Python is uninstalled.)

The module is named ensurepip because if called when pip is already installed, it does nothing. It also has an --upgrade option that will cause it to install the bundled copy of pip if the existing installed version of pip is older than the bundled copy.

enum

The new enum module (defined in PEP 435 ) provides a standard implementation of enumeration types, allowing other modules (such as socket) to provide more informative error messages and better debugging support by replacing opaque integer constants with backwards compatible enumeration values.

pathlib

The new pathlib module offers classes representing filesystem paths with semantics appropriate for different operating systems. Path classes are divided between pure paths, which provide purely computational operations without I/O, and concrete paths, which inherit from pure paths but also provide I/O operations.

For Python 3.4, this module is considered a provisional API.

selectors

The new selectors module (created as part of implementing PEP 3156 ) allows high-level and efficient I/O multiplexing, built upon the select module primitives.

statistics

The new statistics module (defined in PEP 450 ) offers some core statistics functionality directly in the standard library. This module supports calculation of the mean, median, mode, variance and standard deviation of a data series.

tracemalloc

The new tracemalloc module (defined in PEP 454 ) is a debug tool to trace memory blocks allocated by Python. It provides the following information:

• Trace where an object was allocated

• Statistics on allocated memory blocks per filename and per line number: total size, number and average size of allocated memory blocks

• Compute the differences between two snapshots to detect memory leaks

abc

New function abc.get_cache_token() can be used to know when to invalidate caches that are affected by changes in the object graph. (Contributed by Łukasz Langa in bpo-16832 .)

New class ABC has ABCMeta as its meta class. Using ABC as a base class has essentially the same effect as specifying metaclass=abc.ABCMeta, but is simpler to type and easier to read. (Contributed by Bruno Dupuis in bpo-16049 .)

aifc

The getparams() method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in bpo-17818 .)

aifc.open() now supports the context management protocol: when used in a with block, the close() method of the returned object will be called automatically at the end of the block. (Contributed by Serhiy Storchacha in bpo-16486 .)

The writeframesraw() and writeframes() methods now accept any bytes-like object. (Contributed by Serhiy Storchaka in bpo-8311 .)

argparse

The FileType class now accepts encoding and errors arguments, which are passed through to open(). (Contributed by Lucas Maystre in bpo-11175 .)

audioop

audioop now supports 24-bit samples. (Contributed by Serhiy Storchaka in bpo-12866 .)

New byteswap() function converts big-endian samples to little-endian and vice versa. (Contributed by Serhiy Storchaka in bpo-19641 .)

All audioop functions now accept any bytes-like object. Strings are not accepted: they didn’t work before, now they raise an error right away. (Contributed by Serhiy Storchaka in bpo-16685 .)

base64

The encoding and decoding functions in base64 now accept any bytes-like object in cases where it previously required a bytes or bytearray instance. (Contributed by Nick Coghlan in bpo-17839 .)

New functions a85encode(), a85decode(), b85encode(), and b85decode() provide the ability to encode and decode binary data from and to Ascii85 and the git/mercurial Base85 formats, respectively. The a85 functions have options that can be used to make them compatible with the variants of the Ascii85 encoding, including the Adobe variant. (Contributed by Martin Morrison, the Mercurial project, Serhiy Storchaka, and Antoine Pitrou in bpo-17618 .)

collections

The ChainMap.new_child() method now accepts an m argument specifying the child map to add to the chain. This allows an existing mapping and/or a custom mapping type to be used for the child. (Contributed by Vinay Sajip in bpo-16613 .)

colorsys

The number of digits in the coefficients for the RGB — YIQ conversions have been expanded so that they match the FCC NTSC versions. The change in results should be less than 1% and may better match results found elsewhere. (Contributed by Brian Landers and Serhiy Storchaka in bpo-14323 .)

contextlib

The new contextlib.suppress context manager helps to clarify the intent of code that deliberately suppresses exceptions from a single statement. (Contributed by Raymond Hettinger in bpo-15806 and Zero Piraeus in bpo-19266 .)

The new contextlib.redirect_stdout() context manager makes it easier for utility scripts to handle inflexible APIs that write their output to sys.stdout and don’t provide any options to redirect it. Using the context manager, the sys.stdout output can be redirected to any other stream or, in conjunction with io.StringIO, to a string. The latter can be especially useful, for example, to capture output from a function that was written to implement a command line interface. It is recommended only for utility scripts because it affects the global state of sys.stdout. (Contributed by Raymond Hettinger in bpo-15805 .)

The contextlib documentation has also been updated to include a discussion of the differences between single use, reusable and reentrant context managers.

dbm

dbm.open() objects now support the context management protocol. When used in a with statement, the close method of the database object will be called automatically at the end of the block. (Contributed by Claudiu Popa and Nick Coghlan in bpo-19282 .)

dis

Functions show_code(), dis(), distb(), and disassemble() now accept a keyword-only file argument that controls where they write their output.

The dis module is now built around an Instruction class that provides object oriented access to the details of each individual bytecode operation.

A new method, get_instructions(), provides an iterator that emits the Instruction stream for a given piece of Python code. Thus it is now possible to write a program that inspects and manipulates a bytecode object in ways different from those provided by the dis module itself. For example:

>>> import dis
>>> for instr in dis.get_instructions(lambda x: x + 1):
...     print(instr.opname)
LOAD_FAST
LOAD_CONST
BINARY_ADD
RETURN_VALUE

The various display tools in the dis module have been rewritten to use these new components.

In addition, a new application-friendly class Bytecode provides an object-oriented API for inspecting bytecode in both in human-readable form and for iterating over instructions. The Bytecode constructor takes the same arguments that get_instruction() does (plus an optional current_offset), and the resulting object can be iterated to produce Instruction objects. But it also has a dis method, equivalent to calling dis on the constructor argument, but returned as a multi-line string:

>>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3)
>>> for instr in bytecode:
...     print('{} ({})'.format(instr.opname, instr.opcode))
LOAD_FAST (124)
LOAD_CONST (100)
BINARY_ADD (23)
RETURN_VALUE (83)
>>> bytecode.dis().splitlines()       
['  1           0 LOAD_FAST                0 (x)',
 '      -->     3 LOAD_CONST               1 (1)',
 '              6 BINARY_ADD',
 '              7 RETURN_VALUE']

Bytecode also has a class method, from_traceback(), that provides the ability to manipulate a traceback (that is, print(Bytecode.from_traceback(tb).dis()) is equivalent to distb(tb)).

(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in bpo-11816 and Claudiu Popa in bpo-17916 .)

New function stack_effect() computes the effect on the Python stack of a given opcode and argument, information that is not otherwise available. (Contributed by Larry Hastings in bpo-19722 .)

doctest

A new option flag, FAIL_FAST, halts test running as soon as the first failure is detected. (Contributed by R. David Murray and Daniel Urban in bpo-16522 .)

The doctest command line interface now uses argparse, and has two new options, -o and -f. -o allows doctest options to be specified on the command line, and -f is a shorthand for -o FAIL_FAST (to parallel the similar option supported by the unittest CLI). (Contributed by R. David Murray in bpo-11390 .)

doctest will now find doctests in extension module __doc__ strings. (Contributed by Zachary Ware in bpo-3158 .)

email

as_string() now accepts a policy argument to override the default policy of the message when generating a string representation of it. This means that as_string can now be used in more circumstances, instead of having to create and use a generator in order to pass formatting parameters to its flatten method. (Contributed by R. David Murray in bpo-18600 .)

New method as_bytes() added to produce a bytes representation of the message in a fashion similar to how as_string produces a string representation. It does not accept the maxheaderlen argument, but does accept the unixfrom and policy arguments. The Message __bytes__() method calls it, meaning that bytes(mymsg) will now produce the intuitive result: a bytes object containing the fully formatted message. (Contributed by R. David Murray in bpo-18600 .)

The Message.set_param() message now accepts a replace keyword argument. When specified, the associated header will be updated without changing its location in the list of headers. For backward compatibility, the default is False. (Contributed by R. David Murray in bpo-18891 .)

A pair of new subclasses of Message have been added (EmailMessage and MIMEPart), along with a new sub-module, contentmanager and a new policy attribute content_manager. All documentation is currently in the new module, which is being added as part of email’s new provisional API. These classes provide a number of new methods that make extracting content from and inserting content into email messages much easier. For details, see the contentmanager documentation and the email: Examples. These API additions complete the bulk of the work that was planned as part of the email6 project. The currently provisional API is scheduled to become final in Python 3.5 (possibly with a few minor additions in the area of error handling). (Contributed by R. David Murray in bpo-18891 .)

filecmp

A new clear_cache() function provides the ability to clear the filecmp comparison cache, which uses os.stat() information to determine if the file has changed since the last compare. This can be used, for example, if the file might have been changed and re-checked in less time than the resolution of a particular filesystem’s file modification time field. (Contributed by Mark Levitt in bpo-18149 .)

New module attribute DEFAULT_IGNORES provides the list of directories that are used as the default value for the ignore parameter of the dircmp() function. (Contributed by Eli Bendersky in bpo-15442 .)

functools

The new partialmethod() descriptor brings partial argument application to descriptors, just as partial() provides for normal callables. The new descriptor also makes it easier to get arbitrary callables (including partial() instances) to behave like normal instance methods when included in a class definition. (Contributed by Alon Horev and Nick Coghlan in bpo-4331 .)

The new singledispatch() decorator brings support for single-dispatch generic functions to the Python standard library. Where object oriented programming focuses on grouping multiple operations on a common set of data into a class, a generic function focuses on grouping multiple implementations of an operation that allows it to work with different kinds of data.

total_ordering() now supports a return value of NotImplemented from the underlying comparison function. (Contributed by Katie Miller in bpo-10042 .)

A pure-python version of the partial() function is now in the stdlib; in CPython it is overridden by the C accelerated version, but it is available for other implementations to use. (Contributed by Brian Thorne in bpo-12428 .)

gc

New function get_stats() returns a list of three per-generation dictionaries containing the collections statistics since interpreter startup. (Contributed by Antoine Pitrou in bpo-16351 .)

glob

A new function escape() provides a way to escape special characters in a filename so that they do not become part of the globbing expansion but are instead matched literally. (Contributed by Serhiy Storchaka in bpo-8402 .)

hashlib

A new hashlib.pbkdf2_hmac() function provides the PKCS#5 password-based key derivation function 2 . (Contributed by Christian Heimes in bpo-18582 .)

The name attribute of hashlib hash objects is now a formally supported interface. It has always existed in CPython’s hashlib (although it did not return lower case names for all supported hashes), but it was not a public interface and so some other Python implementations have not previously supported it. (Contributed by Jason R. Coombs in bpo-18532 .)

hmac

hmac now accepts bytearray as well as bytes for the key argument to the new() function, and the msg parameter to both the new() function and the update() method now accepts any type supported by the hashlib module. (Contributed by Jonas Borgström in bpo-18240 .)

The digestmod argument to the hmac.new() function may now be any hash digest name recognized by hashlib. In addition, the current behavior in which the value of digestmod defaults to MD5 is deprecated: in a future version of Python there will be no default value. (Contributed by Christian Heimes in bpo-17276 .)

With the addition of block_size and name attributes (and the formal documentation of the digest_size attribute), the hmac module now conforms fully to the PEP 247 API. (Contributed by Christian Heimes in bpo-18775 .)

html

New function unescape() function converts HTML5 character references to the corresponding Unicode characters. (Contributed by Ezio Melotti in bpo-2927 .)

HTMLParser accepts a new keyword argument convert_charrefs that, when True, automatically converts all character references. For backward-compatibility, its value defaults to False, but it will change to True in a future version of Python, so you are invited to set it explicitly and update your code to use this new feature. (Contributed by Ezio Melotti in bpo-13633 .)

The strict argument of HTMLParser is now deprecated. (Contributed by Ezio Melotti in bpo-15114 .)

http

send_error() now accepts an optional additional explain parameter which can be used to provide an extended error description, overriding the hardcoded default if there is one. This extended error description will be formatted using the error_message_format attribute and sent as the body of the error response. (Contributed by Karl Cow in bpo-12921 .)

The http.server command line interface now has a -b/--bind option that causes the server to listen on a specific address. (Contributed by Malte Swart in bpo-17764 .)

idlelib and IDLE

Since idlelib implements the IDLE shell and editor and is not intended for import by other programs, it gets improvements with every release. See Lib/idlelib/NEWS.txt for a cumulative list of changes since 3.3.0, as well as changes made in future 3.4.x releases. This file is also available from the IDLE Help ‣ About IDLE dialog.

importlib

The InspectLoader ABC defines a new method, source_to_code() that accepts source data and a path and returns a code object. The default implementation is equivalent to compile(data, path, 'exec', dont_inherit=True). (Contributed by Eric Snow and Brett Cannon in bpo-15627 .)

InspectLoader also now has a default implementation for the get_code() method. However, it will normally be desirable to override the default implementation for performance reasons. (Contributed by Brett Cannon in bpo-18072 .)

The reload() function has been moved from imp to importlib as part of the imp module deprecation. (Contributed by Berker Peksag in bpo-18193 .)

importlib.util now has a MAGIC_NUMBER attribute providing access to the bytecode version number. This replaces the get_magic() function in the deprecated imp module. (Contributed by Brett Cannon in bpo-18192 .)

New importlib.util functions cache_from_source() and source_from_cache() replace the same-named functions in the deprecated imp module. (Contributed by Brett Cannon in bpo-18194 .)

The importlib bootstrap NamespaceLoader now conforms to the InspectLoader ABC, which means that runpy and python -m can now be used with namespace packages. (Contributed by Brett Cannon in bpo-18058 .)

importlib.util has a new function decode_source() that decodes source from bytes using universal newline processing. This is useful for implementing InspectLoader.get_source() methods.

importlib.machinery.ExtensionFileLoader now has a get_filename() method. This was inadvertently omitted in the original implementation. (Contributed by Eric Snow in bpo-19152 .)

inspect

The inspect module now offers a basic command line interface to quickly display source code and other information for modules, classes and functions. (Contributed by Claudiu Popa and Nick Coghlan in bpo-18626 .)

unwrap() makes it easy to unravel wrapper function chains created by functools.wraps() (and any other API that sets the __wrapped__ attribute on a wrapper function). (Contributed by Daniel Urban, Aaron Iles and Nick Coghlan in bpo-13266 .)

As part of the implementation of the new enum module, the inspect module now has substantially better support for custom __dir__ methods and dynamic class attributes provided through metaclasses. (Contributed by Ethan Furman in bpo-18929 and bpo-19030 .)

getfullargspec() and getargspec() now use the signature() API. This allows them to support a much broader range of callables, including those with __signature__ attributes, those with metadata provided by argument clinic, functools.partial() objects and more. Note that, unlike signature(), these functions still ignore __wrapped__ attributes, and report the already bound first argument for bound methods, so it is still necessary to update your code to use signature() directly if those features are desired. (Contributed by Yury Selivanov in bpo-17481 .)

signature() now supports duck types of CPython functions, which adds support for functions compiled with Cython. (Contributed by Stefan Behnel and Yury Selivanov in bpo-17159 .)

ipaddress

ipaddress was added to the standard library in Python 3.3 as a provisional API. With the release of Python 3.4, this qualification has been removed: ipaddress is now considered a stable API, covered by the normal standard library requirements to maintain backwards compatibility.

A new is_global property is True if an address is globally routeable. (Contributed by Peter Moody in bpo-17400 .)

logging

The TimedRotatingFileHandler has a new atTime parameter that can be used to specify the time of day when rollover should happen. (Contributed by Ronald Oussoren in bpo-9556 .)

SocketHandler and DatagramHandler now support Unix domain sockets (by setting port to None). (Contributed by Vinay Sajip in commit ce46195b56a9.)

fileConfig() now accepts a configparser.RawConfigParser subclass instance for the fname parameter. This facilitates using a configuration file when logging configuration is just a part of the overall application configuration, or where the application modifies the configuration before passing it to fileConfig(). (Contributed by Vinay Sajip in bpo-16110 .)

Logging configuration data received from a socket via the logging.config.listen() function can now be validated before being processed by supplying a verification function as the argument to the new verify keyword argument. (Contributed by Vinay Sajip in bpo-15452 .)

marshal

The default marshal version has been bumped to 3. The code implementing the new version restores the Python2 behavior of recording only one copy of interned strings and preserving the interning on deserialization, and extends this “one copy” ability to any object type (including handling recursive references). This reduces both the size of .pyc files and the amount of memory a module occupies in memory when it is loaded from a .pyc (or .pyo) file. (Contributed by Kristján Valur Jónsson in bpo-16475 , with additional speedups by Antoine Pitrou in bpo-19219 .)

mmap

mmap objects can now be weakrefed. (Contributed by Valerie Lambert in bpo-4885 .)

multiprocessing

On Unix two new start methods, spawn and forkserver, have been added for starting processes using multiprocessing. These make the mixing of processes with threads more robust, and the spawn method matches the semantics that multiprocessing has always used on Windows. New function get_all_start_methods() reports all start methods available on the platform, get_start_method() reports the current start method, and set_start_method() sets the start method. (Contributed by Richard Oudkerk in bpo-8713 .)

multiprocessing also now has the concept of a context, which determines how child processes are created. New function get_context() returns a context that uses a specified start method. It has the same API as the multiprocessing module itself, so you can use it to create Pools and other objects that will operate within that context. This allows a framework and an application or different parts of the same application to use multiprocessing without interfering with each other. (Contributed by Richard Oudkerk in bpo-18999 .)

Except when using the old fork start method, child processes no longer inherit unneeded handles/file descriptors from their parents (part of bpo-8713 ).

multiprocessing now relies on runpy (which implements the -m switch) to initialise __main__ appropriately in child processes when using the spawn or forkserver start methods. This resolves some edge cases where combining multiprocessing, the -m command line switch, and explicit relative imports could cause obscure failures in child processes. (Contributed by Nick Coghlan in bpo-19946 .)

operator

New function length_hint() provides an implementation of the specification for how the __length_hint__() special method should be used, as part of the PEP 424 formal specification of this language feature. (Contributed by Armin Ronacher in bpo-16148 .)

There is now a pure-python version of the operator module available for reference and for use by alternate implementations of Python. (Contributed by Zachary Ware in bpo-16694 .)

os

There are new functions to get and set the inheritable flag of a file descriptor (os.get_inheritable(), os.set_inheritable()) or a Windows handle (os.get_handle_inheritable(), os.set_handle_inheritable()).

New function cpu_count() reports the number of CPUs available on the platform on which Python is running (or None if the count can’t be determined). The multiprocessing.cpu_count() function is now implemented in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, Victor Stinner, and Charles-François Natali in bpo-17914 .)

os.path.samestat() is now available on the Windows platform (and the os.path.samefile() implementation is now shared between Unix and Windows). (Contributed by Brian Curtin in bpo-11939 .)

os.path.ismount() now recognizes volumes mounted below a drive root on Windows. (Contributed by Tim Golden in bpo-9035 .)

os.open() supports two new flags on platforms that provide them, O_PATH (un-opened file descriptor), and O_TMPFILE (unnamed temporary file; as of 3.4.0 release available only on Linux systems with a kernel version of 3.11 or newer that have uapi headers). (Contributed by Christian Heimes in bpo-18673 and Benjamin Peterson, respectively.)

pdb

pdb has been enhanced to handle generators, yield, and yield from in a more useful fashion. This is especially helpful when debugging asyncio based programs. (Contributed by Andrew Svetlov and Xavier de Gaye in bpo-16596 .)

The print command has been removed from pdb, restoring access to the Python print() function from the pdb command line. Python2’s pdb did not have a print command; instead, entering print executed the print statement. In Python3 print was mistakenly made an alias for the pdb p command. p, however, prints the repr of its argument, not the str like the Python2 print command did. Worse, the Python3 pdb print command shadowed the Python3 print function, making it inaccessible at the pdb prompt. (Contributed by Connor Osborn in bpo-18764 .)

pickle

pickle now supports (but does not use by default) a new pickle protocol, protocol 4. This new protocol addresses a number of issues that were present in previous protocols, such as the serialization of nested classes, very large strings and containers, and classes whose __new__() method takes keyword-only arguments. It also provides some efficiency improvements.

plistlib

plistlib now has an API that is similar to the standard pattern for stdlib serialization protocols, with new load(), dump(), loads(), and dumps() functions. (The older API is now deprecated.) In addition to the already supported XML plist format (FMT_XML), it also now supports the binary plist format (FMT_BINARY). (Contributed by Ronald Oussoren and others in bpo-14455 .)

poplib

Two new methods have been added to poplib: capa(), which returns the list of capabilities advertised by the POP server, and stls(), which switches a clear-text POP3 session into an encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo Catucci in bpo-4473 .)

pprint

The pprint module’s PrettyPrinter class and its pformat(), and pprint() functions have a new option, compact, that controls how the output is formatted. Currently setting compact to True means that sequences will be printed with as many sequence elements as will fit within width on each (indented) line. (Contributed by Serhiy Storchaka in bpo-19132 .)

Long strings are now wrapped using Python’s normal line continuation syntax. (Contributed by Antoine Pitrou in bpo-17150 .)

pty

pty.spawn() now returns the status value from os.waitpid() on the child process, instead of None. (Contributed by Gregory P. Smith.)

pydoc

The pydoc module is now based directly on the inspect.signature() introspection API, allowing it to provide signature information for a wider variety of callable objects. This change also means that __wrapped__ attributes are now taken into account when displaying help information. (Contributed by Larry Hastings in bpo-19674 .)

The pydoc module no longer displays the self parameter for already bound methods. Instead, it aims to always display the exact current signature of the supplied callable. (Contributed by Larry Hastings in bpo-20710 .)

In addition to the changes that have been made to pydoc directly, its handling of custom __dir__ methods and various descriptor behaviours has also been improved substantially by the underlying changes in the inspect module.

As the help() builtin is based on pydoc, the above changes also affect the behaviour of help().

re

New fullmatch() function and regex.fullmatch() method anchor the pattern at both ends of the string to match. This provides a way to be explicit about the goal of the match, which avoids a class of subtle bugs where $ characters get lost during code changes or the addition of alternatives to an existing regular expression. (Contributed by Matthew Barnett in bpo-16203 .)

The repr of regex objects now includes the pattern and the flags; the repr of match objects now includes the start, end, and the part of the string that matched. (Contributed by Hugo Lopes Tavares and Serhiy Storchaka in bpo-13592 and bpo-17087 .)

resource

New prlimit() function, available on Linux platforms with a kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the ability to query or set the resource limits for processes other than the one making the call. (Contributed by Christian Heimes in bpo-16595 .)

On Linux kernel version 2.6.36 or later, there are also some new Linux specific constants: RLIMIT_MSGQUEUE, RLIMIT_NICE, RLIMIT_RTPRIO, RLIMIT_RTTIME, and RLIMIT_SIGPENDING. (Contributed by Christian Heimes in bpo-19324 .)

On FreeBSD version 9 and later, there some new FreeBSD specific constants: RLIMIT_SBSIZE, RLIMIT_SWAP, and RLIMIT_NPTS. (Contributed by Claudiu Popa in bpo-19343 .)

select

epoll objects now support the context management protocol. When used in a with statement, the close() method will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in bpo-16488 .)

devpoll objects now have fileno() and close() methods, as well as a new attribute closed. (Contributed by Victor Stinner in bpo-18794 .)

shelve

Shelf instances may now be used in with statements, and will be automatically closed at the end of the with block. (Contributed by Filip Gruszczyński in bpo-13896 .)

shutil

copyfile() now raises a specific Error subclass, SameFileError, when the source and destination are the same file, which allows an application to take appropriate action on this specific error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in bpo-1492704 .)

smtpd

The SMTPServer and SMTPChannel classes now accept a map keyword argument which, if specified, is passed in to asynchat.async_chat as its map argument. This allows an application to avoid affecting the global socket map. (Contributed by Vinay Sajip in bpo-11959 .)

smtplib

SMTPException is now a subclass of OSError, which allows both socket level errors and SMTP protocol level errors to be caught in one try/except statement by code that only cares whether or not an error occurred. (Contributed by Ned Jackson Lovely in bpo-2118 .)

socket

The socket module now supports the CAN_BCM protocol on platforms that support it. (Contributed by Brian Thorne in bpo-15359 .)

Socket objects have new methods to get or set their inheritable flag, get_inheritable() and set_inheritable().

The socket.AF_* and socket.SOCK_* constants are now enumeration values using the new enum module. This allows meaningful names to be printed during debugging, instead of integer “magic numbers”.

The AF_LINK constant is now available on BSD and OSX.

inet_pton() and inet_ntop() are now supported on Windows. (Contributed by Atsuo Ishimoto in bpo-7171 .)

sqlite3

A new boolean parameter to the connect() function, uri, can be used to indicate that the database parameter is a uri (see the SQLite URI documentation ). (Contributed by poq in bpo-13773 .)

ssl

PROTOCOL_TLSv1_1 and PROTOCOL_TLSv1_2 (TLSv1.1 and TLSv1.2 support) have been added; support for these protocols is only available if Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and Antoine Pitrou in bpo-16692 .)

New function create_default_context() provides a standard way to obtain an SSLContext whose settings are intended to be a reasonable balance between compatibility and security. These settings are more stringent than the defaults provided by the SSLContext constructor, and may be adjusted in the future, without prior deprecation, if best-practice security requirements change. The new recommended best practice for using stdlib libraries that support SSL is to use create_default_context() to obtain an SSLContext object, modify it if needed, and then pass it as the context argument of the appropriate stdlib API. (Contributed by Christian Heimes in bpo-19689 .)

SSLContext method load_verify_locations() accepts a new optional argument cadata, which can be used to provide PEM or DER encoded certificates directly via strings or bytes, respectively. (Contributed by Christian Heimes in bpo-18138 .)

New function get_default_verify_paths() returns a named tuple of the paths and environment variables that the set_default_verify_paths() method uses to set OpenSSL’s default cafile and capath. This can be an aid in debugging default verification issues. (Contributed by Christian Heimes in bpo-18143 .)

SSLContext has a new method, cert_store_stats(), that reports the number of loaded X.509 certs, X.509 CA certs, and certificate revocation lists (crls), as well as a get_ca_certs() method that returns a list of the loaded CA certificates. (Contributed by Christian Heimes in bpo-18147 .)

If OpenSSL 0.9.8 or later is available, SSLContext has a new attribute verify_flags that can be used to control the certificate verification process by setting it to some combination of the new constants VERIFY_DEFAULT, VERIFY_CRL_CHECK_LEAF, VERIFY_CRL_CHECK_CHAIN, or VERIFY_X509_STRICT. OpenSSL does not do any CRL verification by default. (Contributed by Christien Heimes in bpo-8813 .)

New SSLContext method load_default_certs() loads a set of default “certificate authority” (CA) certificates from default locations, which vary according to the platform. It can be used to load both TLS web server authentication certificates (purpose=SERVER_AUTH) for a client to use to verify a server, and certificates for a server to use in verifying client certificates (purpose=CLIENT_AUTH). (Contributed by Christian Heimes in bpo-19292 .)

Two new windows-only functions, enum_certificates() and enum_crls() provide the ability to retrieve certificates, certificate information, and CRLs from the Windows cert store. (Contributed by Christian Heimes in bpo-17134 .)

Support for server-side SNI (Server Name Indication) using the new ssl.SSLContext.set_servername_callback() method. (Contributed by Daniel Black in bpo-8109 .)

The dictionary returned by SSLSocket.getpeercert() contains additional X509v3 extension items: crlDistributionPoints, calIssuers, and OCSP URIs. (Contributed by Christian Heimes in bpo-18379 .)

stat

The stat module is now backed by a C implementation in _stat. A C implementation is required as most of the values aren’t standardized and are platform-dependent. (Contributed by Christian Heimes in bpo-11016 .)

The module supports new ST_MODE flags, S_IFDOOR, S_IFPORT, and S_IFWHT. (Contributed by Christian Hiemes in bpo-11016 .)

struct

New function iter_unpack and a new struct.Struct.iter_unpack() method on compiled formats provide streamed unpacking of a buffer containing repeated instances of a given format of data. (Contributed by Antoine Pitrou in bpo-17804 .)

subprocess

check_output() now accepts an input argument that can be used to provide the contents of stdin for the command that is run. (Contributed by Zack Weinberg in bpo-16624 .)

getstatus() and getstatusoutput() now work on Windows. This change was actually inadvertently made in 3.3.4. (Contributed by Tim Golden in bpo-10197 .)

sunau

The getparams() method now returns a namedtuple rather than a plain tuple. (Contributed by Claudiu Popa in bpo-18901 .)

sunau.open() now supports the context management protocol: when used in a with block, the close method of the returned object will be called automatically at the end of the block. (Contributed by Serhiy Storchaka in bpo-18878 .)

AU_write.setsampwidth() now supports 24 bit samples, thus adding support for writing 24 sample using the module. (Contributed by Serhiy Storchaka in bpo-19261 .)

The writeframesraw() and writeframes() methods now accept any bytes-like object. (Contributed by Serhiy Storchaka in bpo-8311 .)

sys

New function sys.getallocatedblocks() returns the current number of blocks allocated by the interpreter. (In CPython with the default --with-pymalloc setting, this is allocations made through the PyObject_Malloc() API.) This can be useful for tracking memory leaks, especially if automated via a test suite. (Contributed by Antoine Pitrou in bpo-13390 .)

When the Python interpreter starts in interactive mode, it checks for an __interactivehook__ attribute on the sys module. If the attribute exists, its value is called with no arguments just before interactive mode is started. The check is made after the PYTHONSTARTUP file is read, so it can be set there. The site module sets it to a function that enables tab completion and history saving (in ~/.python-history) if the platform supports readline. If you do not want this (new) behavior, you can override it in PYTHONSTARTUP, sitecustomize, or usercustomize by deleting this attribute from sys (or setting it to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in bpo-5845 .)

tarfile

The tarfile module now supports a simple Command-Line Interface when called as a script directly or via -m. This can be used to create and extract tarfile archives. (Contributed by Berker Peksag in bpo-13477 .)

textwrap

The TextWrapper class has two new attributes/constructor arguments: max_lines, which limits the number of lines in the output, and placeholder, which is a string that will appear at the end of the output if it has been truncated because of max_lines. Building on these capabilities, a new convenience function shorten() collapses all of the whitespace in the input to single spaces and produces a single line of a given width that ends with the placeholder (by default, [...]). (Contributed by Antoine Pitrou and Serhiy Storchaka in bpo-18585 and bpo-18725 .)

threading

The Thread object representing the main thread can be obtained from the new main_thread() function. In normal conditions this will be the thread from which the Python interpreter was started. (Contributed by Andrew Svetlov in bpo-18882 .)

traceback

A new traceback.clear_frames() function takes a traceback object and clears the local variables in all of the frames it references, reducing the amount of memory consumed. (Contributed by Andrew Kuchling in bpo-1565525 .)

types

A new DynamicClassAttribute() descriptor provides a way to define an attribute that acts normally when looked up through an instance object, but which is routed to the class __getattr__ when looked up through the class. This allows one to have properties active on a class, and have virtual attributes on the class with the same name (see Enum for an example). (Contributed by Ethan Furman in bpo-19030 .)

urllib

urllib.request now supports data: URLs via the DataHandler class. (Contributed by Mathias Panzenböck in bpo-16423 .)

The http method that will be used by a Request class can now be specified by setting a method class attribute on the subclass. (Contributed by Jason R Coombs in bpo-18978 .)

Request objects are now reusable: if the full_url or data attributes are modified, all relevant internal properties are updated. This means, for example, that it is now possible to use the same Request object in more than one OpenerDirector.open() call with different data arguments, or to modify a Request’s url rather than recomputing it from scratch. There is also a new remove_header() method that can be used to remove headers from a Request. (Contributed by Alexey Kachayev in bpo-16464 , Daniel Wozniak in bpo-17485 , and Damien Brecht and Senthil Kumaran in bpo-17272 .)

HTTPError objects now have a