Samuel Adeyinka

I treat this as preparation rather than editing, because each of these operations changes a file in a way that is hard to see and easy to lose. Whitespace is the largest category. Trailing spaces at the end of a line are invisible and appear in diffs as a changed line with nothing visible in it. The tool that removes them is trivial; the judgement is which files they should not be removed from, since trailing whitespace is significant in some formats and in shell scripts. Duplicate lines are the other one. Removing them changes ordering if you sort as part of the operation, so removing duplicates and sorting should be two deliberate steps rather than one button. Keeping the first occurrence rather than the last is a choice that depends on whether the list is a priority order. Encodings produce the failures that look like data problems. A file written with one encoding and read with another produces visible corruption in the extended characters and often a trailing artifact on every row. I cover detecting the encoding and converting explicitly rather than hoping the tool guesses. Line endings are the quiet one. Files with mixed endings are valid and produce diffs where every line appears changed. Empty line handling is a convention rather than a rule: some formats use a blank line as a separator, others treat consecutive blank lines as one, and the two behaviours give different results on the same input. None of these operations is difficult. Each of them is difficult to be confident about afterwards, which is the reason to preview rather than to be careful.