Recipe 04 / operate

Make the boring edges dependable.

The transform is only half the job. simplecsv also covers the common edges around exported files: Windows BOMs, non-comma delimiters, bounded reads, atomic writes, and spreadsheet safety.

Bound reads from untrusted input

ReadCsv reads the complete input into memory. When the source is large or attacker-controlled, use ReadCsvLimit. The record limit includes the header row; a zero limit disables that limit.

file, err := os.Open("upload.csv")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

input, err := simplecsv.ReadCsvLimit(
    file,
    ',',
    10000,              // maximum records, including the header
    10*1024*1024,       // maximum input bytes
)
if err != nil {
    log.Fatal(err)
}

Accept real-world files

All reader and file APIs strip a UTF-8 BOM from the first cell of the first row. A header written by Excel as \ufeffid becomes the expected id. BOMs elsewhere are left as data.

Pass a delimiter when a source uses semicolons or tabs:

input, err := simplecsv.ReadCsvFileComma("export.tsv", '\t')
if err != nil {
    log.Fatal(err)
}

if err := input.WriteCsvFileComma("clean.tsv", '\t'); err != nil {
    log.Fatal(err)
}

Protect spreadsheet users

CSV values beginning with =, +, -, @, tabs, or carriage returns can be interpreted as formulas by spreadsheet applications. The normal write methods preserve values exactly, so sanitization is opt-in.

safe := input.SanitizeFormulas()
if err := safe.WriteCsvFileE("spreadsheet-export.csv"); err != nil {
    log.Fatal(err)
}

SanitizeFormulas returns an independent copy and prefixes risky cells with a single quote. That changes the stored CSV value, so use it when the output is intended for spreadsheet opening, not when exact byte-for-byte data is required.

Write without truncating the destination

WriteCsvFile and WriteCsvFileE write to a temporary file in the destination directory and rename it into place. A failed write does not leave a partially truncated destination. The resulting file uses mode 0600.

Stream to the next command

For Unix pipelines, write directly to stdout:

if err := result.WriteTo(os.Stdout, ','); err != nil {
    log.Fatal(err)
}

Back to all recipes ->