The Data Wrangler's Handbook: Simple Tools for Powerful Results

List of Figures and Tables
Acknowledgments
Introduction

Chapter 1    Getting Started with the Command Line
Finding the Command Line

• Mac
• Windows

Meet the Command Line

Chapter 2    Command Line Concepts
Two Powerful Symbols

• Direct Output to a File (Greater Than Symbol)
• Direct Output to Another Program (Pipe Symbol)

Command Substitution
Regular Expressions—The Swiss Army Knife for Data

• Literal Characters
• Special Characters
• Wildcard Characters
• Logical Operators
• Grouping

Scripting

Chapter 3    Understanding Formats, by David Forero

Chapter 4    Simplify Complicated Problems
Isolating Specific Data Elements
Converting Data into Formats That Are Easier to Work With

Chapter 5    Delimited Text
CSV (Comma Separated Values)

• Commas and Quotation Marks in CSV Files
• Multiline Fields in CSV Files

Multivalued Fields in Delimited Files

Chapter 6    XML
So What Is XML, Really?
What Makes XML So Useful?
Why Is XML So Easy?

• DOM (Document Object Model)
• XPath
• XSLT (eXtensible Stylesheet Language Transformations)

Working with Large XML Files
Working with Complex XML Files
XmlStarlet
Installing XmlStarlet
Converting XML Documents

Chapter 7    JSON (JavaScript Object Notation)

Chapter 8    Scripting
Variables
Arguments
Conditional Execution
Loops

Chapter 9    Solving Common Problems
Viewing Large Files
Locating Files That Contain Particular Data
Finding Files with Specific Characteristics
Working with Internal Metadata
Working with APIs
Combining Data from Different Sources
Other Tasks

Chapter 10    Conclusions
One-Line Wonders
Locating, Viewing, and Performing Basic File Operations

• Combine Information from Multiple Files into a Single File
• Combine Three Files, Each Consisting of a Single Column into a Three-Column Table
• Extract 1,000 Random Lines or Records from a File
• Find Files with Specific Characteristics
• Find All Lines in All Files in the Current Directory as Well as All Subdirectories Containing a Regular Expression
• Identify All Files in Current Directories and Subdirectories That Contain a Value
• List All Files in Current Directory and Subdirectories over a 100 MB in Order of Decreasing Size
• List the Names, Pixel Dimensions, and File Sizes of All Files in the Current Directory and Subdirectories in Tab Delimited Format
• Print Line Number of File That Match Occurred On
• Split Large Files into Smaller Chunks with Each File Breaking on a Line
• View 200 Characters Starting at Position 38562 in a File
• View Lines 4369–4374 of a File

Retrieving and Sending Information over a Network

• Retrieve a Document from the Web and Send It to a File
• Send an XML Document to an API Requiring HTTP Authentication

Sorting, Counting, Deduplication, and File Comparison

• Combine Two Files on a Common Field
• Compare Two Sorted Files
• Count Occurrences for Each Entry in a File, Listed in Order of Decreasing Frequency
• Count Records Containing an Expression
• Count Words, Lines, and Characters in Files
• Identify All Unique Entries and Supply a Count of How Many Times Each Occurs
• Sort a File and Remove Duplicates, Show Only Duplicated Entries, or Show Only Unique Entries

Useful Scripting Operations

• Capture Parameters Passed to a Script
• Divide a Line into Parameters
• Iterate through Every Item in Parameter List
• Perform a Loop
• Perform an Operation Conditionally
• Run a Script on Every Line of a File
• Send the Output of a Command as Arguments to Another Command
• Send the Output of a Command to Another Command
• Send the Output of a Command to a File
• Store the Output of a Command in a Variable
• Use Foreign Character Sets in a Terminal Window

Transforming Text

• Convert File of Dates to YYYY-MM-DD Format
• Convert to Title Case
• Convert to Upper Case
• Convert List of Names from Direct Order to Indirect Order
• Extract and Manipulate All Lines in a File That Match a Complex Pattern
• Extract and Manipulate All Entries in All Files in an Entire Directory Hierarchy That Match a Pattern
• Remove Lines from a File That Match a Pattern
• Remove Carriage Return Characters Inserted by Windows Programs from a File
• Remove Newline Characters from a File
• Replace Newlines in a File with Character 7 (Bell)
• Replace Search_Expr with Replace_Expr Only on Lines That Contain Condition_Expr
• Replace Search_Expr with Replace_Expr Except on Lines That Contain Condition_Expr
• Replace Smart Quotes with Straight Quotes

Working with Delimited Files

• Convert Comma Delimited File Where Some Values Are Quoted and Some Values Are Not to Tab Delimited
• Convert Multiline Records to Table
• Extract Individual Fields from Files
• Find the Most Common Values in the Second Field of a File
• Find All Lines in Tab Delimited File Not Containing Six Fields
• Fix Delimited File That Contains Line Breaks in Fields
• Remove Trailing and Leading Whitespace from Tab Delimited Data Fields
• Reorder Fields in a Tab Delimited File

Working with JSON and XML    

• Add an Attribute to an XML Document
• Add an Element to an XML Document
• Apply XSLT Stylesheet to XML Document
• Convert JSON to Tab Delimited Format
• Delete Elements, Attributes, or Values Based on XPath Expressions
• Display Structure of XML File
• Pretty Print JSON Document
• Pretty Print XML Document

Glossary
Symbols That Perform Important Tasks
Useful Commands
Regular Expression Cheat Sheet
Index