Internet Measurement

Mark Crovella is Professor of Computer Science at Boston University. He has worked in the field of Internet Measurement for more than ten years, and has been a consulant to and founder of Internet technology companies. Balachander Krishnamurthy of AT&T Labs--Research has authored and edited ten books, published more than 70 technical papers, holds 17 patents, and given invited talks in over 30 countries. He co-founded the Internet Measurement Conference and the Steps to Reducing Unwanted Traffic workshop. He has collaborated with over 60 researchers worldwide; his papers are at http://www.research.att.com/~bala/papers/.

• Acknowledgments xxiI Background 11 Introduction 31.1 Why Measure the Internet? 51.2 How to Read this Book 61.3 Resources for More Information 102 Internet Architecture 132.1 The Internet’s Architecture 132.1.1 The History of the Internet 142.1.2 The Organization of the Internet 172.1.3 Design Principles of the Internet 222.2 Details of Internet Operation 252.2.1 Endsystems, Links, and Routers 252.2.2 Autonomous Systems 262.2.3 Routing 262.3 Protocols 312.3.1 IP 322.3.2 TCP 332.3.3 UDP 342.3.4 Routing Protocols 352.3.5 ICMP 352.3.6 SNMP 362.3.7 IP Multicast 372.3.8 DNS 372.3.9 HTTP 392.3.10 P2P 402.4 Applications 423 Analytic Background 453.1 Linear Algebra 453.2 Probability 483.2.1 Background 493.2.2 Special Issues in the Internet 553.3 Statistics 583.3.1 Background 583.3.2 Special Issues in the Internet 613.4 Graphs 643.4.1 Background 643.4.2 Special Issues in the Internet 663.5 Metrics 703.6 Measurement and Modeling 733.6.1 Models in General 733.6.2 The Use of Probability Models 764 Practical Issues in Internet Measurement 794.1 Where can Measurements be Made? 804.1.1 Local Area Network 824.1.2 Inside a Backbone 824.1.3 Entry Points into a Network 844.1.4 Mirror Sites/Network Exchange Points 864.1.5 Wide Area Network 874.2 Role of Time 894.2.1 Background 904.2.2 Sources of Time Information 914.2.3 Synchronized Time 934.3 Role of Internet Directories and Databases 944.3.1 Internet Address and Routing Registries 954.3.2 Domain Name System 974.3.3 Measurement-related Issues in Dealing with Databases 984.4 Measurements Across Various Protocol Layers 994.4.1 Issues in Capturing Data 994.4.2 Changes to Infrastructure/Instrumentation 1024.4.3 Local vs Remote vs Distributed Data Gathering 1034.4.4 Measurement on Overlays 104II In Depth 1055 Infrastructure 1075.1 Properties 1075.1.1 Physical Device Properties 1075.1.2 Topology Properties 1115.1.3 Interaction of Traffic and Network 1125.2 Challenges 1155.2.1 Core Simplicity 1155.2.2 Hidden Layers 1165.2.3 Hidden Pieces 1165.2.4 Administrative Barriers 1175.3 Tools 1175.3.1 Active Measurement 1185.3.2 Passive Measurement 1245.3.3 Fused Measurements 1275.3.4 Bandwidth Measurement 1275.3.5 Latency Measurement and Estimation 1365.3.6 Geolocation 1425.3.7 Inference 1475.3.8 Other Tools 1525.4 State of the Art 1525.4.1 Equipment Properties 1535.4.2 Topology Properties 1545.4.3 Interaction of Traffic and Network 1656 Traffic 1716.1 Properties 1726.1.1 The Basics: Packets and Bytes 1726.1.2 Higher-level Structure 1736.1.3 Flows 1756.1.4 Semantically Distinct Traffic Types 1766.2 Challenges 1766.2.1 Practical Issues 1776.2.2 Statistical Difficulties 1796.3 Tools 1886.3.1 Packet Capture 1886.3.2 Data Management 1916.3.3 Data Reduction 1926.3.4 Inference 2126.4 State of the Art 2156.4.1 Packets and Bytes 2166.4.2 Higher-level Structure 2346.4.3 Flows 2366.4.4 Control Traffic 2386.4.5 Wireless 2397 Applications 2417.1 Application Mix 2427.2 DNS 2447.2.1 DNS Measurement Properties 2457.2.2 DNS Measurement Challenges 2487.2.3 DNS Measurement Tools 2517.2.4 Use of DNS in Other Applications 2567.2.5 State of the Art 2587.3 Web 2697.3.1 Web Measurement Properties 2707.3.2 Web Measurement Challenges 2737.3.3 Web Measurement Tools 2787.3.4 State of the Art 2867.4 P2P 3097.4.1 P2P Measurement Properties 3107.4.2 P2P Measurement Challenges 3147.4.3 P2P Measurement Tools 3177.4.4 State of the Art 3217.5 Online Games 3317.5.1 Games and Measurement Properties 3327.5.2 Networked Games Measurement Challenges 3377.5.3 State of the Art 3407.6 Other Applications 3467.6.1 Streaming Multimedia 346III In Perspective 3538 Anonymization 3558.1 Definitions 3568.2 General Motivation for Anonymizing Data 3578.3 Obstacles and Risks in Sharing Data 3588.4 What Should be Anonymized: Data Categorization 3608.5 How Data is Anonymized: Process and Techniques 3658.5.1 Anonymization Process 3658.5.2 Anonymization Techniques 3678.6 Anonymization Examples at Different Layers 3698.6.1 Configuration Data 3698.6.2 Router-level Data 3708.6.3 Packet-level Traces 3708.6.4 Application-level Data 3738.7 Attacks Against Anonymized Data 3748.8 Anonymizing Data: Metrics for Success 3768.9 Alternatives to Anonymization 3779 Security 3799.1 Role of Internet Measurement in Security 3809.2 Intranet Measurements in Aid of Security 3829.3 Gateway Measurements in Aid of Security 3849.4 Inter-domain Measurements Impact on Security 3869.5 Wide-area Measurements in Aid of Security 3879.6 Application-level Measurements of Attacks 39410 Case Studies 39510.1 Low-level Monitoring Tools 39510.2 Individual Toolsets for Network Measurement 39710.2.1 Windmill 39810.2.2 Click 39910.2.3 dss 40010.2.4 Gigascope 40310.3 Large-scale Measurement Projects 40410.3.1 RIPE 40510.3.2 High-energy Physics 40710.3.3 CAIDA 41010.3.4 PlanetLab 41411 Conclusions and Prospects 41911.1 Trends in Internet Measurement 41911.2 Difficulties 42411.3 Future Work 42611.3.1 Research Challenges 42611.3.2 Emerging Questions 428Bibliography 431Index 473

"The book will be a valuable resource for students and researchers starting to work on internet measurements." (IEEE Network Magazine, May/June 2007)