How to Negotiate With an Agency Over Data Format (2026)

Ask for the format by name, in writing, on day one — “a delimited text export (CSV) or spreadsheet (XLSX), not a PDF or scanned image” — and give the reason in the same letter: the format you need for filtering, sorting, joining and verifying records. If the agency answers with something unusable, you reply in writing, cite the duty to release records in the form they are maintained, and escalate a written reconsideration when the answer does not change. Most format fights are won by having asked precisely, early, and in writing.

That is the whole game. It sounds obvious, and it is still the step most requesters skip, which is why so many newsrooms end up re-keying a PDF by hand.

What follows is the process I watch data reporters use when a records request comes back as a scan: seven steps, the actual letter language, and the escalation ladder for when the first answer is no. Federal US FOIA rules are cited where they apply, and the differences for state sunshine laws and non-US instruments are flagged as we go.

What You Need

Assemble these before you send anything. Most format disputes are won with evidence you already had and did not use.

• Working sample files. A small export from the source system if you can get one, or a screenshot of the published open data portal showing the fields you expect.
• Documentation. Retention schedules, data dictionaries, database inventories, API documentation, published schema files — anything that names the system holding the records.
• Your existing pipeline. If your newsroom already loads a known shape of file, say which tool it is and what breaks without it. That turns a preference into a constraint.
• A named technical owner and a named editorial owner. One person who can answer field-level questions, one who can say what the story needs.
• Written acceptance criteria. The test the delivery has to pass: header row present, one record per row, dates in ISO 8601, no merged cells, a field-level data dictionary alongside it.

The acceptance criteria matter most. “Analysis-ready” is not a specification. “One row per permit, one column per field, no blank rows, no merged headers” is.

Step-by-Step: How to Negotiate With an Agency Over Data Format

1. Define the Format Requirements Your Project Actually Needs

Turn what you want into something testable, field by field. Start with structure: one row per record, one column per field, a header row with stable machine names.

Then data types. Are dates stored as text with slashes or as ISO 8601? Are numbers stored as text with currency symbols? Are codes numeric (state code 17) or spelled out? Each of those is a decision the agency will make for you if you stay silent, and every one of them costs hours downstream.

Write down identifiers too — a stable record ID, permit number, case number. Records without one cannot be deduplicated, and deduplication is what makes a press run or a licensing check trustworthy.

Finally, decide the container: CSV, XLSX, JSON, or a database dump. CSV is the safe default for tabular data. JSON suits nested records that flatten badly. A database dump is worth asking for when you suspect the agency curates its own exports.

2. Gather Evidence Before the Negotiation

Ask the agency questions you already know the answers to, so their reply becomes confirmation rather than a fresh negotiation. Retention schedules are the classic discovery tool: they tell you which system holds the data, who maintains it, and often the record layout itself.

Inspect any file you have already been given. Look for trimmed rows, collapsed categories, silently dropped outliers, rounded figures, dates shifted into a single cell, or pages that begin mid-sequence. Agencies do not usually set out to mislead; the export was built for a meeting, not for analysis.

Write your findings as a short defect list. “4,812 rows expected, 4,790 present, 22 permits missing from 2019 onward” is a fact. “The data looks incomplete” is an opinion, and opinions lose.

3. Ask the Agency for Format Options and Constraints

Ask for options rather than issuing a demand with one door. A good request asks what formats are available, what each one costs to produce, what the delivery schedule looks like, and whether any custom extract is supported.

Then ask why they recommend what they recommend. You often get the constraint for free: a redaction tool that only outputs PDF, a legacy export, a vendor contract that fixes the format.

Here is the clause that does most of the work. Copy it and adjust the bracketed parts:

“Please provide the requested records in a machine-readable electronic format. For tabular records this means a delimited text export such as CSV, or a spreadsheet such as XLSX, with a single header row, one record per row, one field per column, and no merged cells or formatting. PDF and scanned images are not adequate, because they cannot be filtered, sorted, joined or verified without manual transcription of every value. If a different electronic format is available or preferable, please say so and describe its structure, so I can confirm it will work before production begins. I request that any field-level documentation, record layout or data dictionary maintained by the agency be released alongside the data.”

One warning that keeps coming up in public-records communities: do not name PDF and CSV together as if they are interchangeable options. Cheat sheets do this, and it hands the agency a perfectly reasonable way to satisfy you with the one format that solves none of your problems.

4. Compare Each Option Against Shared Criteria

Score each option the agency offers on the same criteria, and write the scores down. Shared criteria turn a preference argument into a comparison.

Format	What you get	Usable for analysis?	Ask for this when
Scanned image or PDF of a printed report	Pixels of a layout	No, unless the table is trivial and OCR is clean	Never as a first choice
PDF with an embedded text layer	Readable text, broken columns, unreliable reading order	Occasionally, for small extracts	Only if nothing else is offered
CSV	Flat delimited text, one row per record	Yes, with a data dictionary	Default for any tabular record set
XLSX	Multiple sheets, formatting, sometimes hidden rows	Yes, after you inspect hidden sheets and formulas	Data is split across logical tables
JSON	Nested records with typed values	Yes, for nested or sparse data	Records have repeating sub-objects
Database dump or SQL export	Native tables, full fidelity	Yes, best case	You suspect curated exports or want every field

Add your own criteria on top: editorial usability, how much manual handling remains, what happens on the next refresh six months from now, accessibility to colleagues who do not code, and the long-term maintenance cost of living with the choice.

5. Trade Scope and Delivery Instead of Arguing About Preferences

Agencies respond to trade. A narrowed scope, a phased production, a later deadline for one field, an agreement to supplement later — each of these is currency you can spend to get the format you actually need.

Useful trades include: accepting a partial release now with a written commitment on the remainder; taking the first batch as XLSX while a CSV is built; agreeing to pay duplication costs if a custom extract is required; or accepting a slower schedule in exchange for field-level documentation.

The trade to refuse is giving up the machine-readable requirement. Accepting a PDF with no date for the real file is how a request quietly dies.

When an agency is genuinely constrained — a mainframe system that only prints, a redaction vendor that outputs PDF, a system nobody has queried in years — say so in your file. Ask what it would take to fix the pipeline, and ask for the raw extract even if unredacted field names must be withheld. You can work around a missing column. You cannot work around a missing system.

6. Document the Agreement in Testable Terms

When the agency agrees, capture the agreement where both sides can find it: the format, the file extension, the structure, the permitted variations, the schema version, the delivery cadence, the error-handling contact, the responsibilities on each side, and the approval step.

Say what happens when it goes wrong. Who notices a broken delivery, who tells whom, and within how many days. Errors in a machine-readable file are often invisible until someone publishes a wrong number, so detection needs an owner.

Also decide, in advance, what a change costs. A schema change should require a version bump and a notice period. Without that, you will rebuild your parser twice a year.

7. Validate the First Delivery and Set a Change Process

Test the first delivery the day it arrives, not the week before publication. Count rows against your expectation. Open the file and check for merged cells, stray formulas, hidden rows, blank headers, inconsistent date formats, and encoding problems with accented characters.

Record every failure as a numbered defect and send them as one list rather than drip-feeding complaints. Defect lists get fixed; separate emails get answered.

Then confirm acceptance in writing, and set the change process: how you request a fix, how you request a new version, and how the agency tells you about a planned schema change. A single named data steward or open data officer on the agency side saves months.

Common Mistakes

Most of these are recoverable in the first letter, and unrecoverable in the third exchange.

• Negotiating before you know what you need. Fix: write the acceptance criteria and a small representative sample first. Agree your requirements internally before you send a word.
• Treating a sample file as a production guarantee. Fix: specify that the production delivery must match the agreed structure, and get the sample’s field list confirmed in writing.
• Accepting vague field descriptions. “Date” does not say whether it is a filing date, an event date or an entry date. Fix: require a field-level data dictionary or record layout with the delivery.
• Leaving versioning out. Fix: require a schema version in a sidecar file or a header row, plus advance notice before a change.
• Never testing edge cases. Unusual characters, dates before 1970, negative amounts, ten-digit identifiers that a spreadsheet silently converts to scientific notation. Fix: ask for the outlier rows specifically.
• Conflating format refusal with records validity. If the agency says records do not exist, that is a different dispute with a different appeal route than a dispute over how they are released. Fix: separate the two issues in writing so neither is lost behind the other.
• Arguing that one format is simply better. Fix: argue from burden and statute. The manual re-entry cost and the duty to release records in the form maintained are both harder to argue with than taste.

Negotiation Tips for Newsroom and Agency Teams

Keep the exchange collaborative without going soft. Separate must-haves from preferences in your own notes, so you concede cheaply on the second list and hold the first.

Use shared examples. A short anonymised extract of what you plan to do with the data answers “why do you need this?” better than any argument about interoperability. Keep all written decisions in one place with dates, so nobody re-litigates a settled point. Assign one technical owner and one editorial owner on each side, and schedule a short call after each written exchange — ten minutes on the phone resolves more format questions than a week of email.

If you work outside the United States, the shape of the process holds but the citations change. In the UK, a FOIL request sits under the Freedom of Information Act 2000, with the Information Commissioner’s office as the appeal route; Australia uses FOI Acts per jurisdiction with an external review body; many countries publish an open data or re-use directive that pushes agencies toward machine-readable formats by default. Check which instrument actually applies before you cite anything, and note the jurisdiction in your own file so a colleague does not reuse your argument where it does not hold.

Frequently Asked Questions

Conclusion: Start With Testable Requirements

Knowing how to negotiate with an agency over data format comes down to four moves: name the format on day one, give a concrete reason tied to analysis, point at the duty to produce records electronically, and escalate in writing when the first answer is no.

Start with the smallest useful piece of work: write a one-page requirements checklist your side agrees on, and put together a small representative sample of what you expect the file to look like. Everything after that gets easier.