Effective Python

Brett Slatkin is a Principal Software Engineer at Google in the Office of the CTO, focusing on emerging technologies. He co-founded Google Surveys, launched Google Cloud’s first product (App Engine), and co-created the PubSubHubbub protocol—all using Python. Brett has been writing Python code professionally for the past 19 years and has made numerous contributions to open-source projects.

• Preface     xviiAcknowledgments     xxiiiAbout the Author     xxv  Chapter 1: Pythonic Thinking     1Item 1: Know Which Version of Python You’re Using     1Item 2: Follow the PEP 8 Style Guide     3Item 3: Never Expect Python to Detect Errors at Compile Time     6Item 4: Write Helper Functions Instead of Complex Expressions     8Item 5: Prefer Multiple-Assignment Unpacking Over Indexing     11Item 6: Always Surround Single-Element Tuples with Parentheses     16Item 7: Consider Conditional Expressions for Simple Inline Logic     19Item 8: Prevent Repetition with Assignment Expressions     24Item 9: Consider match for Destructuring in Flow Control; Avoid When if Statements Are Sufficient     30  Chapter 2: Strings and Slicing     41Item 10: Know the Differences Between bytes and str     41Item 11: Prefer Interpolated F-Strings over C-Style Format Strings and str.format     47Item 12: Understand the Difference Between  repr and str when Printing Objects     58Item 13: Prefer Explicit String Concatenation over Implicit, Especially in Lists     62Item 14: Know How to Slice Sequences     67Item 15: Avoid Striding and Slicing in a Single Expression     70Item 16: Prefer Catch-All Unpacking Over Slicing     72  Chapter 3: Loops and Iterators     77Item 17: Prefer enumerate over range     77Item 18: Use zip to Process Iterators in Parallel     79Item 19: Avoid else Blocks After for and while Loops     82Item 20: Never Use for Loop Variables After the Loop Ends     85Item 21: Be Defensive when Iterating over Arguments     87Item 22: Never Modify Containers While Iterating over Them; Use Copies or Caches Instead     92Item 23: Pass Iterators to any and all for Efficient Short-Circuiting Logic     98Item 24: Consider itertools for Working with Iterators and Generators     102  Chapter 4: Dictionaries     109Item 25: Be Cautious when Relying on Dictionary Insertion Ordering     109Item 26: Prefer get over in and KeyError to Handle Missing Dictionary Keys     117Item 27: Prefer defaultdict over setdefault to Handle Missing Items in Internal State     122Item 28: Know How to Construct Key-Dependent Default Values with __missing__     124Item 29: Compose Classes Instead of Deeply Nesting Dictionaries, Lists, and Tuples     127  Chapter 5: Functions     135Item 30: Know That Function Arguments Can Be Mutated     135Item 31: Return Dedicated Result Objects Instead of Requiring Function Callers to Unpack More Than Three Variables     138Item 32: Prefer Raising Exceptions to Returning None     142Item 33: Know How Closures Interact with Variable Scope and nonlocal     145Item 34: Reduce Visual Noise with Variable Positional Arguments     150Item 35: Provide Optional Behavior with Keyword Arguments     153Item 36: Use None and Docstrings to Specify Dynamic Default Arguments     157Item 37: Enforce Clarity with Keyword-Only and Positional-Only Arguments     161Item 38: Define Function Decorators with functools.wraps     166Item 39: Prefer functools.partial over lambda Expressions for Glue Functions     169  Chapter 6: Comprehensions and Generators     173Item 40: Use Comprehensions Instead of map and filter     173Item 41: Avoid More Than Two Control Subexpressions in Comprehensions     176Item 42: Reduce Repetition in Comprehensions with Assignment Expressions     178Item 43: Consider Generators Instead of Returning Lists     182Item 44: Consider Generator Expressions for Large List Comprehensions     184Item 45: Compose Multiple Generators with yield from     186Item 46: Pass Iterators into Generators as Arguments Instead of Calling the send Method     188Item 47: Manage Iterative State Transitions with a Class Instead of the Generator throw Method     195  Chapter 7: Classes and Interfaces     201Item 48: Accept Functions Instead of Classes for Simple Interfaces     201Item 49: Prefer Object-Oriented Polymorphism over Functions with isinstance Checks     205Item 50: Consider functools.singledispatch for Functional-Style Programming Instead of Object-Oriented Polymorphism     210Item 51: Prefer dataclasses for Defining Lightweight Classes     217Item 52: Use @classmethod Polymorphism to Construct Objects Generically     230Item 53: Initialize Parent Classes with super     235Item 54: Consider Composing Functionality with Mix-in Classes     240Item 55: Prefer Public Attributes over Private Ones     245Item 56: Prefer dataclasses for Creating Immutable Objects     250Item 57: Inherit from collections.abc Classes for Custom Container Types     260  Chapter 8: Metaclasses and Attributes     265Item 58: Use Plain Attributes Instead of Setter and Getter Methods     265Item 59: Consider @property Instead of Refactoring Attributes     270Item 60: Use Descriptors for Reusable @property Methods     274Item 61: Use __getattr__, __getattribute__, and __setattr__ for Lazy Attributes     279Item 62: Validate Subclasses with __init_subclass__     285Item 63: Register Class Existence with __init_subclass__     293Item 64: Annotate Class Attributes with __set_name__     299Item 65: Consider Class Body Definition Order to Establish Relationships Between Attributes     303Item 66: Prefer Class Decorators over Metaclasses for Composable Class Extensions     310  Chapter 9: Concurrency and Parallelism     319Item 67: Use subprocess to Manage Child Processes     320Item 68: Use Threads for Blocking I/O; Avoid for Parallelism     324Item 69: Use Lock to Prevent Data Races in Threads     330Item 70: Use Queue to Coordinate Work Between Threads     333Item 71: Know How to Recognize When Concurrency Is Necessary     344Item 72: Avoid Creating New Thread Instances for On-Demand Fan-out     349Item 73: Understand How Using Queue for Concurrency Requires Refactoring     353Item 74: Consider ThreadPoolExecutor When Threads Are Necessary for Concurrency     361Item 75: Achieve Highly Concurrent I/O with Coroutines     364Item 76: Know How to Port Threaded I/O to asyncio     368Item 77: Mix Threads and Coroutines to Ease the Transition to asyncio     381Item 78: Maximize Responsiveness of asyncio Event Loops with async-friendly Worker Threads     389Item 79: Consider concurrent.futures for True Parallelism     393  Chapter 10: Robustness     399Item 80: Take Advantage of Each Block in try/except/else/finally     399Item 81: assert Internal Assumptions and raise Missed Expectations     404Item 82: Consider contextlib and with Statements for Reusable try/finally Behavior     408Item 83: Always Make try Blocks as Short as Possible     412Item 84: Beware of Exception Variables Disappearing     414Item 85: Beware of Catching the Exception Class     416Item 86: Understand the Difference Between Exception and BaseException     419Item 87: Use traceback for Enhanced Exception Reporting     424Item 88: Consider Explicitly Chaining Exceptions to Clarify Tracebacks     428Item 89: Always Pass Resources into Generators and Have Callers Clean Them Up Outside     436Item 90: Never Set __debug__ to False     442Item 91: Avoid exec and eval Unless You’re Building a Developer Tool     445  Chapter 11: Performance     447Item 92: Profile Before Optimizing     448Item 93: Optimize Performance-Critical Code Using timeit Microbenchmarks     453Item 94: Know When and How to Replace Python with Another Programming Language     458Item 95: Consider ctypes to Rapidly Integrate with Native Libraries     462Item 96: Consider Extension Modules to Maximize Performance and Ergonomics     467Item 97: Rely on Precompiled Bytecode and File System Caching to Improve Startup Time     475Item 98: Lazy-Load Modules with Dynamic Imports to Reduce Startup Time     478Item 99: Consider memoryview and bytearray for Zero-Copy Interactions with bytes     485  Chapter 12: Data Structures & Algorithms     493Item 100: Sort by Complex Criteria Using the key Parameter     493Item 101: Know the Difference Between sort and sorted     499Item 102: Consider Searching Sorted Sequences with bisect     501Item 103: Prefer deque for Producer-Consumer Queues     504Item 104: Know How to Use heapq for Priority Queues     509Item 105: Use datetime Instead of time for Local Clocks     519Item 106: Use decimal When Precision Is Paramount     523Item 107: Make pickle Serialization Maintainable with copyreg     526  Chapter 13: Testing and Debugging     533Item 108: Verify Related Behaviors in TestCase Subclasses     533Item 109: Prefer Integration Tests over Unit Tests     541Item 110: Isolate Tests From Each Other with setUp, tearDown, setUpModule, and tearDownModule     547Item 111: Use Mocks to Test Code with Complex Dependencies     550Item 112: Encapsulate Dependencies to Facilitate Mocking and Testing     559Item 113: Use assertAlmostEqual to Control Precision in Floating Point Tests     563Item 114: Consider Interactive Debugging with pdb     565Item 115: Use tracemalloc to Understand Memory Usage and Leaks     570  Chapter 14: Collaboration     575Item 116: Know Where to Find Community-Built Modules     575Item 117: Use Virtual Environments for Isolated and Reproducible Dependencies     576Item 118: Write Docstrings for Every Function, Class, and Module     582Item 119: Use Packages to Organize Modules and Provide Stable APIs     588Item 120: Consider Module-Scoped Code to Configure Deployment Environments     593Item 121: Define a Root Exception to Insulate Callers from APIs     595Item 122: Know How to Break Circular Dependencies     600Item 123: Consider warnings to Refactor and Migrate Usage     605Item 124: Consider Static Analysis via typing to Obviate Bugs     613Item 125: Prefer Open Source Projects for Bundling Python Programs over zipimport and zipapp     621  Index     627