World Wide Webber - Steve Vinoski, Building RESTful Web Services with Erlang and Yaws

Steve Vinoski, Building RESTful Web Services with Erlang and Yaws

Posted: 23 November 2008 @ 23:19 UT from San Francisco, USA

Here are my raw, unedited notes from Steve Vinoski's QCon San Francisco 2008 talk.

- Got into REST around 2001/2002 because of Mark Baker
- Erlang overview - functional, concurrent, distributed, focus on reliability; small language
- YAWS - Yet Another Web Server - good for serving dynamic content; embeddable, or stand alone
- Dynamic content - embed <erl> tags in HTML in a .yaws file
- Out function - ties your apps to the Web server - an arg record gives connection details to the application - a big tuple
- Ehtml avoids embedding <erl> tags which get messy, instead write HTML with Erlang syntax - tuple space of tuples
- Obligatory Emacs rant...
- Appmods - application module exports a function, is bound to one or more URIs via configuration file. Yaws calls Out function on any appmods registered for a URI.
- Question: Can you bind appmods to multiple URI elements? Answer: Yes. Also one appmod to multiple URIs too.
- Yapps - Yaws applications - a whole Erlang application (usually a collection of modules) with reliability characteristics in mind. Can be restarted etc. Lives in the same VM, safely.
- Yapps are tied to URIs just the same as appmods. Typically bind Yapp early in the URI, and appmods later
- Yapp framework is optional - uses mnesia, the fault tolerant filestore by default (requires config). Can configure Yapps to bootstrap with other stores. Steve created in-memory store with same interface as mnesia.
- Design URIs well - should be easy on the eye, because that makes your server implementation better
- Erlang pattern matching, allows functional overloading based on number and value of arguments. Underscored arguments are optional (compiler warns otherwise)
- Pattern matching supports URI-based dispatch - look in the Arg and break out the path. Split path into list, pass the path and URI into the out/3 function - generalise to add specific handlers for each URI path
- Question: is pattern matching on URIs using the pattern common in Erlang? Answer: Yes, it probably is. Could also use a table to dispatch on whole URI
- URI Design allows you to take advantage of the language and pattern match. So friendly URIs buy you a lot
- Representations and media types - resources can have many representations (see Content-Type header), and clients can use Accept header to negotiate (list of preferences using q parameters)
- Joe Gregorio wrote a mimeparse library in Python for this, Steve recently ported it to Erlang
- Using mimeparse in Erlang simply pass a list of mime types you want, from the Accept header and it returns the best match of the acceptable types, or an empty list
- If you can't match a type based on the accept header, then return 406 Not Acceptable
- create out/4 function which takes Arg, Uri, mime type, order and let pattern matching do its thing
- Question: Why not just add .xml etc to the end of your URI? Answer: You can go either way.
- Yaws has a JSON module, or Mochiweb has a one too. Erlang has xmerl (part of Erlang) or erlsom (add on package, modern faster than xmerl)
- Question: Can you grab mochiweb without grief? Answer: There are modules that are separable that you can pull in.
- For multiple representations, use overloaded functions for out/4 function for the resource
- Vary header tells caches that representation will change depending on the content type
- Yaws gives you a lot of power for not much code
- HATEOAS - many RESTful APIs miss this part of REST. XML has xlink to help, json making progress
- Comment: realistically xlink not widely implemented
- Comment: there's something called "JSON reference"
- Each resource decided what methods it supports, get the method from the Arg record coming into your Out function
- Question on methods changing over resource lifecycle - TBD
- HTTP methods are dispatched based on pattern matching again - drag the method out of the Arg record
- Yaws has functions for extracting query and post data (versus extracting POST data as binary from Arg record)
- Conditional GET is critical to scalability - use ETags and last-modified headers - return them as a header from Yaws just like any other header, a tuple of {header-name, value}
- Etags takes precedence over last modified if both are sent
- Yaws is stable and robust; Erlang/OTP under the covers, so can load code for yapps and appmods on the fly.
- Question: Are you using Erlang in your job? Answer: Yes, both Erlang and Yaws
- Monitoring (heart) for watching process, supervisor in process too. Can run 10,000s processes cheaply (1M in 0.5 seconds).
- Launch a process, link to it, if it dies you'll get a notification (even over networks)
- Yaws has an interactive mode - Steve has never started the debugger in Erlang!
- OPTIONS support recently added, sendfile support added in C via linked-in driver, testing being improved
- Question: Yaws versus WebMachine? Answer: Some people like other frameworks

comment: 1

Comments:

Great notes, Jim, just a few clarifications:

- Ehtml doesn't eliminate the need for erl tags; the two are orthogonal. The tags are used to run Erlang code and have its results placed into a page, while ehtml is used to write HTML constructs that are effectively native Erlang. Ehtml can be used within code within erl tags.

- Just to be clear for your readers, my "Emacs rant" was praising Emacs. :-)

- Underscored Erlang arguments are unused within the function and are ignored by the compiler. If you name an argument normally but don't use it, the compiler warns that it's unused, and a leading underscore prevents that.

- The in-memory store I created doesn't replace Mnesia generally; what I did was create an in-memory version of the module that the Yapp capability uses to access Mnesia. I did this because I bootstrap all my yapps through yaws.conf and thus have no need to persistently remember them in Mnesia.

- Regarding the question about methods changing over the resource lifetime: presumably state related to the resource is available to the server so it can tell what methods are allowed at any given time. If a method is called that isn't allowed, you return a 405. You can also support OPTIONS on the resource and have it return the list of methods that resource supports at the instant OPTIONS is called.

- Regarding my comment about the debugger: the point here is that due to Erlang's REPL, its trace capabilities which can be manipulated as needed in the REPL, and due to Erlang supporting the loading of new module versions on the fly, I've never had the need to run a debugger when developing with Erlang.

- And finally, regarding other Erlang web frameworks like Mochiweb, Nitrogen, and webmachine, I think they all add value and choosing the right one depends on what you're trying to do. A lot of people like Mochiweb, for example, while the other two mentioned here are newer. Many of the techniques covered in my talk apply to the other frameworks too.

Posted by Steve Vinoski, email, url @ 24 November 2008 04:19 UT