Our promo site is powered by C#, ASP.NET Webpages and three TeamDesk databases holding the data for it – Database Library, Testimonials and Experts. As we recently finished site redesign, unifying look and feel for various site parts, we've also switched from SOAP-based data retrieval to REST API. While in this article we'll share our findings concentrating on Experts as an example – as this part not only reads the data but posts new leads into the database – many pieces of the code are available for reuse as they are capable to deal with any application and any type of data.
Planning Functionality
As we want search engines to index the content, we are performing all data operations server-side – we extract the information from Experts database via the API and render it as a part of the page to pretend search engine it is a static content. Next, we do not want to bother the database to return fresh set of data on each and every page view – while REST API supports conditional caching, experts update their information infrequently. In order to decrease API workload and speed up page rendering we are caching information used often in memory for next 10 minutes unconditionally and only then perform conditional call to retrieve the data.
Next, we've switched "Contact Expert" form from sending data directly to database via web-to-record functionality to send the data to promo site first. The site performs basic validation for mandatory fields and e-mail address correctness and displays messages if data validation has not passed. Also this approach would enable us to extend the functionality with, say, CAPTCHA validation. Only when the data is completely valid we send it to a database via REST API.
Dealing with JSON
To deal with JSON input and output we've chosen excellent Newtonsoft.Json library – it has reach capabilities, good throughput and tons of options to tweak every piece of serialization and deserialization process. The library is able to serialize and deserialize JSON both from and to .NET dynamic types and to C# classes. We have created model classes for both Experts and Leads as with strongly-typed approach we can minimize typos and take advantage of IntelliSense in the IDE. In most cases names of class members match the column names in the database. In case column name cannot be expressed as a member identifier, thanks to the library, mapping between JSON name and member name can be controlled via member's attribute.
public class Expert
{
public string Id;
public string Photo;
public string Name;
public string Location;
[JsonProperty("TeamDesk Experience")]
public string Experience;
public string PR;
public string HR;
public string User;
public string Description;
public bool Overbooked;
}
public class Lead
{
// Reference to User in Experts
public string Expert;
public string Name;
[JsonProperty("E-Mail")]
public string Email;
public string Question;
}
Sending and receiving
To send and receive the data we are utilizing .NET 4.5 HttpClient class. HttpClient builds the request message and parses the response, but actual data interchange logic is performed via message transport class. We want the transport to be able to receive compressed content and cache it but default message transport due to portability issues supports only compression options. In order to support caching we need to use another transport class. It is desktop-specific, but it's ok for us. We wrote small function that sets up HttpClient with all the options we need:
public static class ApiHelper
{
// …
static HttpClient CreateClient()
{
return new HttpClient(
new WebRequestHandler() {
AutomaticDecompression = DecompressionMethods.GZip
| DecompressionMethods.Deflate
});
}
// …
}
Caching support is completely transparent – as long as data is in the cache and server reports cached copy is still valid via HTTP 304 status code HttpClient returns cached copy with HTTP 200 OK status as the request was actually made.
Caching
Caching facility provided by HttpClient allows us to minimize the traffic between the site and the database but API calls are still performed to ensure the validity of cached data. As experts' information is changed infrequently we can trade freshness for reducing the number of API calls. To do this we organize in-memory cache to keep parsed API response for 10 minutes. Once entry is missing or was removed from the cache we'll call API again and place the result back to the cache shifting expiration time for 10 more minutes. Here is another small helper class with a single Get method. The method is generic and is capable to cache any output type under any key type.
public static class Cache
{
public static TValue Get<TKey, TValue>(TKey key, Func<TKey, TValue> loader)
where TValue : class
{
string cacheKey = typeof(Cache).FullName + "$" +
typeof(TValue).FullName + "$" +
key.ToString();
TValue result = MemoryCache.Default.Get(cacheKey) as TValue;
if(result == null)
{
result = loader(key);
result = MemoryCache.Default.AddOrGetExisting(cacheKey, result,
DateTimeOffset.Now + Timeout) as TValue ?? result;
}
return result;
}
static readonly TimeSpan Timeout = TimeSpan.FromMinutes(10);
}
Generic Data Retrieval
Our ApiHelper class has another couple methods to help dealing with select API calls. As our data retrieval tasks are not limited to Experts, the Select method is generic to deserializing the output into a list of objects of any type. Any unsuccessful HTTP status code results to an exception which is handled globally. SelectOne is a shorthand method returning first record in the list, if any. Literal method encodes the quotes inside the text and adds quotes around for use in filter conditions.
public static class ApiHelper
{
// …
public static string Literal(string value)
{
if(value == null) return "null";
return String.Concat('"', value.Replace(@"\", @"\\").Replace(@"""", @"\"""), '"');
}
public static IEnumerable<T> Select<T>(
string apiPath,
string table,
IEnumerable<string> columns = null,
string filter = null,
IEnumerable<string> sort = null,
int top = 0, int skip = 0)
{
// apiPath is in format:
// https://www.teamdesk.net/secure/api/v2/<app-id>/<token>
// Build full REST API URL including parameters
StringBuilder apiUrl = new StringBuilder(apiPath)
.Append('/')
.Append(Uri.EscapeDataString(table))
.Append("/select.json");
char sep = '?';
if(columns != null && columns.Any())
{
apiUrl.Append(sep).Append(String.Join("&",
columns.Select(c => "column=" + Uri.EscapeDataString(c))));
sep = '&';
}
if(!String.IsNullOrEmpty(filter))
{
apiUrl.Append(sep).Append("filter=")
.Append(Uri.EscapeDataString(filter));
sep = '&';
}
if(sort != null && sort.Any())
{
apiUrl.Append(sep).Append(String.Join("&",
sort.Select(c => "sort=" + Uri.EscapeDataString(c))));
sep = '&';
}
if(top != 0)
{
apiUrl.Append(sep).Append("top=").Append(top);
sep = '&';
}
if(skip != 0)
{
apiUrl.Append(sep).Append("skip=").Append(skip);
sep = '&';
}
// Call the API
using(var client = CreateClient())
{
// return a list of objects of type T
return JsonConvert.DeserializeObject<IEnumerable<T>>(
client
.GetAsync(apiUrl.ToString())
// will throw on failure
.Result.EnsureSuccessStatusCode()
.Content.ReadAsStringAsync().Result);
}
}
public static T SelectOne<T>(
string apiPath,
string table,
IEnumerable<string> columns = null,
string filter = null)
{
return Select<T>(apiPath, table, columns, filter).FirstOrDefault();
}
// …
}
Retrieving Experts
Enough with generic helpers, now let's do some actual work. On experts page we render short info as tiles, while the page dedicated to an expert contains much longer description and a contact form if expert is available. Let's create Experts class – Service property retrieves base path to API from configuration file, List method with no parameters retrieves (probably in-memory cached copy of) the list of experts to render tiles, Get method retrieves single expert information to render the page – in-memory caching is not used in this case. GetPhotoURL converts filename;revision;guid value reported for attachment columns into an URL to attachment REST API method.
public static class Experts
{
// …
internal static string Service
{
// https://www.teamdesk.net/secure/api/v2/<app-id>/<token>
get { return ConfigurationManager.AppSettings["Experts.REST"]; }
}
// Retrieves the list of experts – either via cached copy or via API call
public static IEnumerable<Expert> List()
{
return Cache.Get("Experts", _ =>
ApiHelper.Select<Expert>(
Service,
"Expert", new string[] {
"Id", "Photo ", "Name", "Location",
"PR", "HR", "TeamDesk Experience"
},
"[Active]",
Enumerable.Repeat("Date Modified//DESC", 1)));
}
// Retrieves the expert by key, does not cache in memory
public static Expert Get(string key)
{
return ApiHelper.SelectOne<Expert>(
Service,
"Expert", new string[] {
"Id", "Photo ", "Name", "Location",
"PR", "HR", "TeamDesk Experience",
"Description", "Overbooked", "User"
},
"[Id]=" + ApiHelper.Literal(key));
}
// Get URL to /Expert/Photo/attachment/<guid> method
// (Public Access is on for Photo column)
public static string GetPhotoURL(string value)
{
// strip off authorization, add table, field and method
return value != null ?
Experts.Service.Substring(0, Experts.Service.Length – 33) +
"/Expert/Photo/attachment/" +
value.Substring(value.Length – 36) : null;
}
// …
}
With help of Experts class page rendering looks like:
<article>
@foreach(var expert in Experts.List())
{
<section>
<img src="@Experts.GetPhotoURL(expert.Photo)" />
<h3>@expert.Name</h3>
<p>from @expert.Location</p>
<table>
<tr><th>Starting Project Rate</th><td>@expert.PR</td></tr>
<tr><th>Hourly Rate</th><td>@expert.HR</td></tr>
</table>
<ul>
@foreach(var exp in expert.Experience.Split(','))
{
<li class="ui-icon-check">@exp</li>
}
</ul>
<a href="~/experts?id=@expert.Id">View expert profile</a>
</section>
}
</article>
Creating Expert's Leads
Let's add one more method to ApiHelper class – the method to create single record. Create API method responds with individual status for every row passed in and while using strongly typed data for input we are taking advantage of parsing the output as a dynamic object – just as a proof of concept.
public static class ApiHelper
{
// …
public static void CreateOne<T>(string apiPath, string table, T data)
{
string apiUrl = String.Format("{0}/{1}/create.json",
apiPath,
Uri.EscapeDataString(table));
using(var client = CreateClient())
{
dynamic result = JsonConvert.DeserializeObject(
client
.PostAsync(
apiUrl,
new StringContent(
JsonConvert.SerializeObject(
Enumerable.Repeat(data, 1)),
Encoding.UTF8,
"application/json"))
.Result.EnsureSuccessStatusCode()
.Content.ReadAsStringAsync().Result);
if(result[0].status != 201) // not created? Throw!
throw new InvalidOperationException(
(string)result[0].errors[0].message);
}
}
// …
}
Another method in Experts class fills Lead model from a form and calls ApiHelper
public static class Experts
{
// …
public static void AddLead(string expert, string name,
string email, string question)
{
ApiHelper.CreateOne(Experts.Service, "Lead", new Lead() {
Expert = expert,
Name = name,
Email = email,
Question = question
});
}
// …
}
And on the page we are doing the following.
@{
Expert expert = Experts.Get(Request.QueryString["id"]);
// …
// Set up form validation rules
// …
if(IsPost && Validation.IsValid() && !expert.Overbooked)
{
try
{
Experts.AddLead(
expert.User,
Request.Form["name"],
Request.Form["email"],
Request.Form["question"]);
Response.Redirect("~/experts");
return;
}
catch(Exception ex)
{
Validation.AddFormError(ex.Message);
}
}
}
<!– Render expert page –>
Resume
In this article we shared some techniques we've used to power our promo-site with REST API. While we do not demand to follow our model precisely as there might be some restrictions, or in contrast, some technologies that may require or make possible to use different approaches to the same task, we hope you've found our samples useful – either in form of reusable code or ideas on code and data organization.
