Publishing to your open data portal automatically

Open data portals start with good intentions. A dataset of permits or service requests goes up, people use it, and then it quietly goes stale because someone has to export and upload it by hand. A dataset last updated eight months ago does more harm than good.

Automating the refresh is straightforward. Most of the work is deciding what should be public in the first place.

Start with a privacy review

Before any automation, sit down with the data owner and your records or privacy officer. Go field by field.

• Clearly public: permit number, type, status, issue date, valuation, general location.
• Needs a decision: exact address, contractor name, free-text descriptions.
• Not public: applicant phone and email, reporter names, internal notes.

Free-text fields deserve extra care. A 311 description can contain a name, a phone number or a medical detail. Many cities leave free text out of open datasets entirely.

Also decide which records to drop altogether, such as requests that involve minors, domestic situations or certain case types.

Choose where it goes

Socrata-based portals

Many city portals run on Socrata. Its SODA API lets a program read and write dataset rows. If each row has a row ID, such as the permit number, writes become upserts: a new permit is added, a changed one is updated, and nothing is duplicated.

ArcGIS Hub

ArcGIS Hub sites usually share feature layers from ArcGIS Online or Enterprise. Keeping the Hub dataset current means keeping the feature layer current: add new features and update changed ones on a schedule.

A live data API

Some audiences want live data rather than a nightly dataset. A read-only API that answers from the system of record, with paging and rate limits, can sit alongside the portal.

The flow, step by step

A typical nightly publishing flow looks like this:

• On a schedule. Nightly is a common choice for permits and requests.
• Get records, only changed. Read records changed since the last successful run, not the whole history each night.
• Map fields. Send only the fields you approved in the privacy review. Anything not mapped never leaves.
• Send records. Upsert into the dataset by row ID.

Because only mapped fields are sent, adding a field to the source system doesn't accidentally publish it. Someone has to add it to the map on purpose.

Avoid the common problems

Duplicates

Without a stable row ID, every run appends the same records again. Pick an ID that never changes, such as the permit or request number, and use it for every write.

Deleted or withdrawn records

Decide what happens when a permit is withdrawn or a request is marked as a duplicate. Options include updating its status, or removing the row. Write the decision down.

Coordinates

Map-ready datasets need latitude and longitude in a consistent system. If the source holds State Plane coordinates, convert them before publishing. Check that points actually fall inside your city.

Dates and time zones

Publish dates in a consistent format and say which time zone they are in. Portals and spreadsheet users read them differently.

Write the data dictionary once

Every public dataset should have a short description of each field, how often it updates, and what is excluded. It answers most questions from journalists and residents before they are asked. Update it whenever the field map changes.

Watch it after launch

• Alert when a nightly run reads no records, which can mean the source changed.
• Alert when volume jumps or drops sharply against recent runs.
• Check the dataset's "last updated" date on a regular schedule.

In SentraBridge

SentraBridge connects to Socrata portals with upserts by row ID, and to ArcGIS feature layers for Hub. A template publishes OpenGov permits nightly, and any read-only flow can also be published as a live data API. Rows pass through in memory and aren't stored. See the open data solution.