This example will show how to listen for a Wufoo form’s webhook, and push the resulting data into Socrata. Socrata offers embeddable forms for all tabular datasets, but integrating with Wufoo will allow you to take advantage of their powerful form-building GUI and validation settings.
Wufoo has the ability to enable a webhook that fires every time a user submits a form. A webhook is just an HTTP POST to a URL specified by the form administrator, and it carries a form-encoded payload of the data that was entered into the form.
To get the data from this webhook into Socrata, we’ll use a very simple node.js app running on heroku. The app will have a single POST endpoint whose sole purpose in life is to listen for the wufoo webhook. When the webhook is triggered, the app will transform the data into a format that can be pushed to the SODA Publisher API. POSTing this data will add a new row to the target dataset!
Let’s start by setting up a dataset that had the columns we want. For this demo, we’ll be collecting data for upcoming events. Each event will have a title, start date, end date, location name, address, and description. All of the fields will be plain text except for the dates.
Next, we create a new form on Wufoo that mirrors the columns we set up in the Socrata dataset. The form has single-line text input for title and location, two date selectors, a set of address inputs (
country), and a multi-line text input for description. Once the form is complete, it is ready to use immediately and will log data in Wufoo. Now it’s time to turn on the webhook!
Under the Wufoo form manager screen, there’s a link for “Notifications”.
On the next screen, there’s an option to send notifications “to another application”. We’ll choose webhook from the dropdown and click “Add integration”.
Now we have to tell Wufoo what URL to POST to when the form is submitted. Eventually this will be our node app, but to get an idea of what the POST looks like, we can use a neat little service called requestBin. RequestBin just gives us a URL that we can POST to, and it shows a nice breakdown of the request in the browser. Create a requestbin URL and paste it into the webhook URL field in our new Webhook notification.
Now, if we fill out and submit the form…
We’ll see the details of the POST appear in the requestBin!
Look in the “Form/Post Parameters” section, you’ll see our precious form data ready to be parsed, transformed, and sent to Socrata!
The node app is middle-man. It listens for data coming from the Wufoo webhook. When it receives a POST, it transforms the data a bit to fit the Socrata dataset, then sends it on its way by POSTing the transformed data to Socrata. (Read on assuming we’ve changed our Wufoo webhook URL to the public-facing url of this node app)
For our purposes, I won’t cover every detail of this app, but here are some of the modules in use:
A single POST endpoint is all we need. It simply takes the request body (that bodyParser has already converted into an object), runs it through
transformPayload(), pushes the resulting transformed data to an Array, then passes the array to
transformPayload() maps the form fields to keys in a new object that match our Socrata dataset. For example, our event title gets POSTed from Wufoo as
Field1. For the POST to Socrata to work, the title must have a key of
event_title. Since the form’s address field is actually several text inputs, we concatenate them into a single string.
We also to a little moment.js magic to convert the YYYYmmdd dates that came from Wufoo into valid ISO8601 date strings that Socrata expects.
The initial form data looked like this:
… and the result of transformData() looks like this:
The resulting object gets pushed to its own single-item array, and is then ready to POST!
postToSocrata() does the honors:
If everything worked properly, you should see the following response from the SODA API in the server logs, and the data should appear as a new row in your Socrata dataset!
There you have it! Webhooks allow us to not waste precious resources polling endpoints to look for new form submissions. We can pass along the data directly to Socrata, where it is publicly accessible and discoverable in the catalog!
Check out the code on Github. Happy Hacking!