Text Processing (Grep/Sed/Awk)

Stop manually stitching CSV files with weird line breaks

🧩 The Challenge

Dealing with CSV exports from legacy accounting software that randomly injects newlines into fields is a nightmare. I spent three hours last week writing a Python script just to realize it could have been one line of awk.

💡 The Fix

Use awk to identify lines that don’t match your expected field count and treat them as continuations of the previous record. It keeps your data pipeline moving without writing custom parsers.

awk -F, 'NF < 5 {printf "%s ", $0; next} 1' input.csv > cleaned.csv

⚙️ Why It Works

By setting the record separator logic to check for the field count, you effectively merge the mangled line with the one that preceded it before printing. Simple state management in the stream is always faster than loading the whole file into memory.

🚀 Pro-Tip: Swap that 5 for the exact column count of your source file to catch every single broken row.

Linux Tips & Tricks | © ngelinux.com | 9/22/2026