Wonder what webhooks are?
Integration. That’s the slogan of modern web. Nowadays, information is stored in the cloud. Web servers handle your emails, contacts, invoices and what not and you are not bound to a single service provider. Store contacts here, write emails there, manage your business everywhere else.
But how your email provider can use contacts stored on another web server? There are couple of options.
It can retrieve full list of contacts every time you need to select one from the list. Of course there is a drawback: the larger the list, the more time it takes to retrieve.
It can store a copy of contact list and periodically query whether there are new/updated contacts since last synchronization attempt. Again, the question is: how often sync operation should happen? And how often you add or update your contacts? Whatever the period is you are either at risk of missing updates or running bytes over the network just to find out nothing happened. This process called polling.
But what if your contact manager web site will notify all interested parties upon any change in contact list? Webhooks concept is to help.
Webhooks-enabled web service defines events, something that happens on their side, and maintains the list of subscribers, URLs of other web sites it will send event information to. With webhooks you do not need to poll any longer, you simply listen for, say, “Hello there, new contact is added”.
For a long time polling was an only option in TeamDesk without the use of third-party integration sites. No longer! Today we are introducing webhooks support.
Webhooks in TeamDesk
Remember, webhooks are concept, not a standard. There are many properties webhooks from different providers have in common, yet there are some things to treat specifically. There are also some constraints we have to comply to.
Request Format
POST request seems a standard, but content format varies. While JSON format is widespread, some services use forms, and we even encountered some exotic cases where JSON payload is stored at one of the form’s fields.
Timing
You should process the data and respond back within certain timeframe or you’ll be unsubscribed. Timeframe is commonly set to 30 seconds, but some services are quite restrictive – Intuit Quickbooks wants you to respond in 3 seconds. Failure to respond leads to eventual removal from the list of subscribers.
Special responses
While many web services expect response with just an OK status, some require responding with certain text (HelloSign), some issue specially crafted requests and expect specially crafted response to validate the subscription (Microsoft Graph, DropBox, Xero).
Signatures
And finally, endpoint is a publicly facing URL anyone can send any data to. IP address of the sender is not the protection — cloud-hosted services can use wide range of IP addresses. The answer is the signature.
Based on the content and the secret key known only to you and web service they calculate some unique text – signature. When you get the data you should perform the same process and compare the signature you calculated with the signature they provided. Signature validation is not required but serves as an additional security measure.
How do we handle it?
Strict timing means we can’t run any database-specific logic when data arrives – there is no way to guarantee processing will complete in required timeframe. So, we’ll validate request format, verify the signature if any, store the data in our internal queue and respond to the caller. At this point we are done with time critical part.
Next, record in the queue is transformed to a new record in a regular TeamDesk table. As every webhook has its own format of incoming data we create one table per webhook. We’ll try to process the queue as soon as possible, but to keep the chronology of incoming data, one webhook processes one queue entry at the time; processing of new item in the queue won’t start until all previous items are processed. Using regular TeamDesk table will let you run validation rules, change triggers and actions providing you with the chance to reject the data or update records in other tables.
Ok, let’s see it in action.
As an example, we’ll transform our HelloSign Integration database to use webhooks instead of polling. Let’s outline current process.
We send the document to HelloSign for signature. HelloSign notifies all the parties the action is needed. Some may respond within minutes, for some it may take hour and days to respond. When all parties signed, the database should download signed document. Current implementation uses time-dependent trigger to query HelloSign once in an hour for status updates. Yet, HelloSign is webhook-enabled service and we can speed up the process by listening to HelloSign signature-related events and update status and download the document almost instantly.
Let’s start by creating new webhook.
New Webhook
Navigate to Setup | Database tab | Tools | Webhooks, click New button.
By default your new webhook is configured to most commonly used options and is ready to save.
Endpoint: the piece that follows your-database-url/hooks/ is generated randomly by default and unique for a database, though you can provide some meaningful name.
Name: the name of new table webhook data will be stored to.
Notes: any text you want to attribute your webhook with.
Sender: first three options denote the data format webhook expects to receive when no special processing is required. Other options are services that either require some special responses or need a shared secret to calculate the signature. If the service you are integrating with requiring either please drop us a note at support@teamdesk.net, we’ll be happily add it to the list.
Now let’s fully configure one for HelloSign.
Step 1. New webhook for HelloSign
First, you have to have an account with HelloSign, sign in and navigate to HelloSign API Settings. Reveal and copy API Key (1).
Then create new webhook in TeamDesk. Important is to select HelloSign as Sender as it requires some special processing and signature verification. As HelloSign signature algorithm uses API key as a shared secret, paste copied HelloSign API key to Shared Secret field. Also, we’ll give the webhook some easy to remember URL and name the table to store the data.
Save the webhook. Copy the endpoint link by clicking on it and paste into HelloSign API Settings Account Callback field (2). Save HelloSign API Settings. Done with this step.
As webhook setup requires actions on both TeamDesk and third-party provider side, we designed webhook setup as a staged process.
Step 2. Wait for incoming data
Now webhook waits for incoming data. This will let you check if webhook is working in general, and collect some data samples.
Let’s trigger events by sending the document through a full signature cycle and refresh.
Once you get the data successfully, yellow banner will be replaced with Assignments section.
Pending status of the record in the log means the data is in the queue, but not yet processed – you do not have assignments yet and running processing will simply create an empty record in webbook’s backing table.
Step 3. Setting up assignments
Our database workflow is interested in two status changes – either all parties are signed and signed document is ready to download, or at least one party is declined. In HelloSign case, every event payload is uniform and provides full information about the signature process, we simply need to capture a couple of fields out of event payload: event_type, signature_request_id (as it uniquely identifies document record sent for signature), is_complete and is_declined. First two are of text type, latter two are boolean/checkbox.
Let’s create assignments.
Step 4. Workflow logic
With assignments set up we’ll have event data we are interested in as records in HelloSign Events table. Now we need to set up the mechanism to update document’s status and download the file.
Let’s add workflow logic to the HelloSign Event table. When you send document for signature, signature request id is assigned to HelloSign Request ID column of the Documents table. HelloSign Event table also has HelloSign Request ID captured, let’s create the many-to-many relation between these two tables and match records by HelloSign Request ID and add Recordset column to it. This column will allow us to update matching document record from HelloSign Event record.
Now let’s add a trigger
And an update action to a trigger:
Ok, now setup is complete. It’s time to let webhook process pending events to see if it works correctly.
And after some time the queue gets processed
And final step is to visit document record to ensure its status is Complete and file is downloaded 🙂
Other applications
Some web services that perform potentially lengthy operations (for example CloudConvert API v1 and Stripe Checkout) accept URLs to call when operation is complete or cancelled (callback URLs). These URLs are not exactly webhooks. They are one-time runners and do not require upfront registration, you provide them as a part “start operation” payload. But you can still use TeamDesk webhooks to process incoming data.
