The Great Yesod EventSource Experiment of September 2012

This post is merely about getting an EventSource app running in Yesod. For the JavaScript parts, I refer you to two articles:

Those are the two sources I have been using myself. Regarding the chapter in the Yesod book on the matter of EventSources, I think that particular example could have been somewhat clearer and more direct if there hadn't been several concepts involved at the same time (subsite, EventSource, the wiki thing). Hence this post, that in essence strips away everything but the EventSource skeleton. I have taken the liberty to put a little flesh on its clean-picked bones, however, and use three div's into which I write the data of three different events. This is just to exemplify how this is done both in the client side script and in the server handler.

There's also a new package called yesod-eventsource out. I'm sure it's great, but I couldn't wrap my head around it in 5 minutes since there are no working examples. I therefore went with using Wai's EventSource instead just as in the Yesod book.

But, I digress. The code is available in the Github repo https://github.com/obscaenvs/yest, aswell as here for your benefit:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
{-# LANGUAGE OverloadedStrings, TypeFamilies, QuasiQuotes,
  TemplateHaskell, FlexibleInstances, MultiParamTypeClasses,
  FlexibleContexts #-}


import Yesod
import Yesod.Form.Jquery
import Control.Concurrent.Chan (Chan, dupChan, writeChan, newChan)
import Control.Concurrent (forkIO, threadDelay)
import Data.Text (Text, pack)
import Network.Wai.EventSource (ServerEvent (..), eventSourceAppChan)
import Blaze.ByteString.Builder.Char.Utf8 (fromText, fromString)
import Data.Monoid (mappend)

data ESrc = ESrc (Chan ServerEvent)

-- just a var
pageTitleText = "The Great Yesod EventSource Experiment of September 2012" :: Text

mkYesod "ESrc" [parseRoutes|
/recv ReceiveR GET
/setup SetupR GET
|]

instance Yesod ESrc where
    defaultLayout w = do
                      y <- getYesod
                      p <- widgetToPageContent (w `mappend` addScriptEither (urlJqueryJs y))
                      mmsg <- getMessage
                      hamletToRepHtml [hamlet| $newline never
$doctype 5

<html>
    <head>
        <title>#{pageTitle p}
        ^{pageHead p}
    <body>
        $maybe msg <- mmsg
            <p .message>#{msg}
        ^{pageBody p}
|]


instance YesodJquery ESrc where
    urlJqueryJs _ = Right "http://ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js"

getReceiveR :: Handler ()
getReceiveR = do
  ESrc chan0 <- getYesod
  chan <- liftIO $ dupChan chan0
  req <- waiRequest
  res <- lift $ eventSourceAppChan chan req
  sendWaiResponse res

getSetupR :: Handler RepHtml
getSetupR = do
    defaultLayout $ do
      setTitle $ toHtml pageTitleText
      eventSourceW

eventSourceW = do
  receptacle0 <- lift newIdent -- css id for output div 0
  receptacle1 <- lift newIdent -- css id for output div 1
  receptacle2 <- lift newIdent -- css id for output div 2
  [whamlet| $newline never
            <div ##{receptacle0} .outdiv>^^Unclassified output up here.
            <div ##{receptacle1} .outdiv>^^Output 1 up here.
            <div ##{receptacle2} .outdiv>^^Output 2 up here.|]
  toWidget [lucius|
           .outdiv
           {
             float:left;
             width:400px;
             font-family:courier,'courier new',sans-serif;
           }
         |]
  toWidget [julius|
                    // setup the EventSource itself
                    var source = new EventSource('/recv');
                    // listener for first type of events
                    source.addEventListener('evstr1', function(event) {
                                                               $('##{receptacle1}').prepend('EventStream 1: <strong>' +
                                                               event.data + ' </strong><br>');          
                                                         }, false);
                    source.addEventListener('evstr2', function(event) {
                                                               $('##{receptacle2}').prepend('EventStream 2: <strong>' +
                                                               event.data + ' </strong><br>');          
                                                         }, false);
                    source.onmessage = function (event) {
                      // a message without a type was fired
                      $('##{receptacle0}').prepend('Event: <strong>' +
                      event.data + ' </strong><br>');
                    };
            |]

talk :: Chan ServerEvent -> Int -> IO ()
talk ch n = do
  writeChan ch $ ServerEvent Nothing (Just $ fromString $ show n) $
            return $ fromText "Hello World! " `mappend` fromString (show n)
  threadDelay micros
  writeChan ch $ ServerEvent (Just $ fromText "evstr1") (Just $ fromString $ show n) $
            return $ fromText "Hello World 1! " `mappend` fromString (show n)
  threadDelay micros
  writeChan ch $ ServerEvent (Just $ fromText "evstr2") (Just $ fromString $ show n) $
            return $ fromText "Hello World 2! " `mappend` fromString (show n)
  threadDelay micros
  talk ch (n+1)
    where micros = (1*(10^6))

main = do
    ch <- newChan
    forkIO $ talk ch 0
    warpDebug 3000 $ ESrc ch

Some comments about this:

  • Line 24:defaultLayout is defined. Actually, there is only one tiny difference from Yesod's default one: the inclusion of a call to addScriptEither. This is done in order to get the version of jQuery that we desire; c.f. line 43.
  • Line 43: we override Yesod's default Google CDN URL with a slightly more modern one.
  • Line 46:getReceiveR is the handler resposible for sending stuff on the wire to the client. In order to do so, we duplicate the channel boxed inside the ESrc foundation data type, into which talk is writing data. For more on this, consult the documentation for Control.Concurrent.Chan and Network.Wai.EventSource.
  • Line 95: talk uses the ServerEvent data constructor directly—yes, it's actually exposed to the world. For the full story on how to use it, see the API doc at Network.Wai.EventSource.
  • Lines 99, 102, 105: threadDelay is used to pause the forked thread that talk is running in in order not to flood the listening clients with data. It takes a number of microseconds as argument.

With Yesod version 1.0 or 1.1, this should just work.

Yesod and subsites: a no-brainer

Update 20130316: As of Yesod 1.2, this post will be obsolete since the way Yesod is treating subsites is changed. See Michael Snoyman's blog post on the matter.

I am following two main threads in my Haskelling these days:

  • immersing myself in the Yesod web framework, and
  • immersing myself in Concurrent/Parallel Haskell.

The reason for these parallel (pardon the pun) threads (ouch, again!) being I am writing a HTML5 web app where I have to use a bit of Concurrent Haskell on the server side.

For the Yesod part of things, I have invested in a dead tree copy of the Yesod Book and I am following it quite diligently. Now, for this particular day I needed to create two subsites, referenced from a master. A bare bones implementation looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
{-# LANGUAGE QuasiQuotes, TypeFamilies, MultiParamTypeClasses #-}
{-# LANGUAGE TemplateHaskell, FlexibleInstances, OverloadedStrings #-}
import Yesod

data SubSite1 = SubSite1
data SubSite2 = SubSite2

getSub1RootR :: Yesod master => GHandler SubSite1 master RepHtml
getSub1RootR = defaultLayout [whamlet|Welcome to subsite 1!|]

getSub2RootR :: Yesod master => GHandler SubSite2 master RepHtml
getSub2RootR = defaultLayout [whamlet|Welcome to subsite 2!|]

mkYesodSub "SubSite1" [] [parseRoutes|
/ Sub1RootR GET
|]

mkYesodSub "SubSite2" [] [parseRoutes|
/ Sub2RootR GET
|]

data Master = Master
    {
      getSub1 :: SubSite1
    , getSub2 :: SubSite2
    }

mkYesod "Master" [parseRoutes|
/ RootR GET
/sub1 SubSite1R SubSite1 getSub1
/sub2 SubSite2R SubSite2 getSub2
|]

instance Yesod Master

getRootR :: GHandler sub Master RepHtml
getRootR = defaultLayout [whamlet|
<h1>Welcome to the master page
<p>
    <a href=@{SubSite1R Sub1RootR}>sub1
<p>
    <a href=@{SubSite2R Sub2RootR}>sub2
|]

main = warpDebug 3000 $ Master SubSite1 SubSite2

Were this post to end here, it would have been even weaker than it is. However, there is an anecdote attached to it that illustrates the convenience of the strong static typing of Haskell, and this is the real intention of the post: at first, I only added /sub2 SubSite2R SubSite2 getSub2 in the parseroutes argument for mkYesod (line 28), but failed to add SubSite2 to the data constructor invocation for Master in main (line 45). In my rushed state I had mistakenly read that one data constructor invocation as two separate type constructor invocations for the data types and consequently as theirs being individual arguments to warpDebug — which made me think I couldn't possibly reference the other subsite there. Now, I compiled to see what I got so I could figure out where to stick a reference to the second subsite. Guess what? Compilation error! And it kindly told me there was stuff missing in the data constructor call... I immediately realised what a half-wit I was, and that that Master call was a data constructor. Having corrected the constructor invocation I compiled again and it worked. In a language with weaker typing it would probably have run, missing one route, and leaving me flabbergasted for yet some time.

This is admittedly a silly mistake even by my standards, but silly mistakes happen all the time in programming. Here at least, I am relieved to say, I had a friend in the compiler protecting me against my own idiocy. I will cry a deluge the day I have to go back using weakly typed languages...

A resource post on Concurrent and Parallel Haskell

At long last, this past summer I have been dipping my head in the swamp that is concurrency and parallelism in Haskell. In doing so, I cannot help but feel a noticeable lack in the way of comprehensive learning material. Chapter 24 in Real World Haskell certainly goes a long way, but some of the explanations need a little elaborating on — the paragraphs treating seq and par immediately spring to mind. With this in mind, I decided to compile a set of links and comments that might help smooth out the learning curve for anybody trying to get into the area. This post will most likely be updated several times in the near and far future, so check back or — even better — suggest additions and changes.

Prerequisites

In my opinion, in order to understand parallelism properly, you first need a better understanding of laziness and evalution to WHNF ("weak head normal form"). Of course, there are probably lots of research papers on the subject, but I have read none of these (if you have any suggestion as to a good paper readable for mere mortals, drop me a line). A few resources that are available on the subject in the form of easily accessed online tutorials and articles are these:

  • The laziness chapter in the Haskell Wikibook: unfinished work, but there's some good stuff in it. I'd recommend skipping everything after and including the header "Benefits of nonstrict semantics", though, it being poorly written and largely unfinished.
  • Laziness on the Haskell wiki: An implementation of merge sort is used as an example to elucidate the difference between a strict and a lazy function. Educational, if only for giving the insight that "There are some operations that cannot be done lazily, for instance, sorting a list".
  • Lazy evaluation tutorial on the Haskell wiki: can be seen as an introductory explanation of laziness.

Articles and books

  • Real World Haskell, Chapter 24: the starting point, after the prerequisites are met. The chapter uses the old Strategy datatype, and contains no material about the Eval or Par monads.
  • Real World Haskell, Chapter 28: this can probably wait until you have learned about the traditional way of doing concurrent programming. I have yet to trawl through it myself.
  • the chapter on concurrency in the Haskell Wikibook: wholly unfinished work; skip it.
  • Upcoming: O'Reilly book on Parallel and Concurrent Haskell. Simon Marlow will be the author, and a preliminary date for release is March 2013. According to this, the book will be

    a significantly revised and extended version of the Parallel and Concurrent Haskell tutorial from CEFP'11.

    For notes on the tutorial and a link to it, see its entry under the heading "Papers" below.

Papers

  • "Runtime Support for Multicore Haskell" (PDF): this paper is mentioned in the documentation of Control.Parallel.Strategies on Hackage. The description of a Strategy in Real World Haskell is outdated since the old approach led to space leaks when the GHC garbage collector was updated. It is not explained in great detail on Hackage just how the new approach works; this very accessible paper does a better job of not only that, but explains the ins and outs of parallelism in Haskell in general. It made me realise the importance of tuning the garbage collector to achieve good performance. Highly recommended.
  • Parallel and Concurrent Haskell tutorial from CEFP'11 (PDF): it weighs in at a well-written 63 pages, and uses the new Evaluation Strategies-concept by means of the Eval monad. That's about what I know now, I have just started reading it. It does seem, however, that this is the definitive tutorial for now, since it uses the new API.

Blog posts

Lectures and talks

  • Simon Peyton Jones: Data Parallel Haskell: SPJ in action. There's an interesting bit in the beginning about functional programming and parallelism in the 80's, and why it failed to deliver in those olden days.

General notes and thoughts

  • The Eval monad replaces the old use of par and seq.
  • seq is a language primitive/atom, i.e. it is built into the compiler (read:"GHC") itself. Consequently, you cannot browse the source on Hackage at http://hackage.haskell.org/packages/archive/base/latest/doc/html/Prelude.html#v:seq. I think this should really — I mean really — be mentioned early in any decent account of parallelism. I found this particular piece of info in a comment on StackOverflow!
  • What x `seq` y really does is evaluate x by means of "compiler magic" to see if it is not undefined, then letting the value of the expression be y if this is the case, and undefined if not. The description of seq in Real World Haskell really should mention this too.
  • On "Head Normal Form" (HNF): the Wikibook chapter on laziness mentioned above tells us

    There is also a 'head normal form', but it's not used in Haskell.

    Real World Haskell, on the other hand, tells us — incorrectly — that

    The familiar seq function evaluates an expression to what we call head normal form[...]

    This could have been a mere blunder, but in the next paragraph the authors say quite clearly that there is a difference between WHNF and HNF. This all, of course, prompts the casual observer to pose the question "WTF?". According to this excellent StackOverflow entry, though, the question seems settled as there being no HNF in Haskell, making Real World Haskell erroneous on this topic.

A clarification on the subject of "MultiParamTypeClasses" and functional dependencies

Update 20130316: In the Mezzo Haskell book chapter on type classes on GitHub a similar path of explaining MultiParamClasses and FunctionalDependencies is taken, where the notion of converting things is also used as an example. I would like to stress that I in no way plagiarised that chapter, but came up with this notion independently of that book. It's such a simple example, though, so it probably jumps at anyone who wants to do an example for MultiParamTypeClasses. I heartily recommend the aforementioned chapter as it is more thorough than this post.

The Haskell Wiki page for functional dependencies is by and large good, but I think it wouldn't hurt to stress this: the functional dependency isn't automatically derived by any magical means; it is you, the programmer, that chooses what the dependency should be.

Let me illustrate by a simple type class that captures the notion that something can be converted to something else:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{-# LANGUAGE MultiParamTypeClasses,
    FunctionalDependencies, FlexibleInstances #-}

class Convertible s t | s -> t where
    convert :: s -> t

instance Convertible Char String where
    convert c = [c]

{-
--impossible to add since a functional dependency is defined
--for this type class, and we have already chosen our dependency:
instance Convertible Char Char where
    convert = id
-}


{- this *is* possible due to MultiParamTypeClasses being used -}
class Convertible' s t where
    convert' :: s -> t

instance Convertible' Char Char where
    convert' = id

instance Convertible' Char String where
    convert' c = [c]

The instances of Convertible' work, because there is no functional dependency defined for the type class; we do not have to choose between the two. On the other hand, we have to specify the type desired when using convert' on a Char.

Another point that I think deserves mentioning is that the order of the types matters:

1
2
3
4
5
6
7
8
9
10
{-# LANGUAGE MultiParamTypeClasses,
    FunctionalDependencies, FlexibleInstances #-}

class FooClass s t u | s t -> u where
    fooFunc :: s -> t -> u

instance FooClass Char String Char where
    fooFunc c s = c

instance FooClass String Char String where
    fooFunc s c = s

GHC will be able to infer the correct version of fooFunc to use without an additional type specification since the two instances do not overlap.

Looking back at the Haskell Wiki page for functional dependencies, note that it is thus possible to overload on right and left multiplication of a matrix with a vector (but see [2] in the notes below):

1
2
instance Mult Matrix Vector Vector where (...)
instance Mult Vector Matrix Vector where (...)

    Notes:

  1. Concerning the need for FlexibleInstances, I refer you to Real World Haskell, chapter 15, section "Rounding out our module", and to the GHC manual. Be warned, though, the material in the manual is rather obscure.
  2. It could be argued that these really should be

    1
    2
    instance Mult Matrix Vector Vector where (...)
    instance Mult CoVector Matrix CoVector where (...)

    Opting for this would naturally make moot the overloading of right and left multiplication.

  3. Functional dependencies is not overly well documented; the GHC manual even says so itself.

Lenses

There seems to be great things afoot in the Haskell community, more specifically in the lens camp. Don't know what a lens is? Here's the perfect introduction, by Gabriel Gonzalez: Haskell for all: 'Lenses'. It also has the merit of being rather recently posted.

At Planet Haskell a few blog posts about more powerful versions of this concept have caught my attention, the source URL:s of these being

The lens packages fclabels and data-lens are not particularly friendly for the new user since they are lacking a basic 'how-to', but the introduction by Gabriel Gonzalez goes a long way at remedying this. There is more material coming up from him, he says, but he wants to

wait for the more recent lens discussions on reddit to settle down because it looks like more powerful versions are on the way.

I can only assume the Reddit discussion is connected to the above comonad.com and r6.ca blog posts since I can't be bothered following a long discussion on Reddit for the time being.

It's an interesting time to be a Haskeller, even if it's of the wannabe variant — yes, reference to self intended. On the other hand, it could be argued that it's always interesting to be a Haskeller...

A projective conundrum

So today I was learning how to use the Applicative type class. I am working through 'Using Parsec', chapter 16 of Real World Haskell. Reading up a bit on the Wikibook entry and correcting a blatant error there, I soon instead found myself hoogling *>. The definition for this is to be found in Control.Applicative, f below being a functor:

1
2
(*>) :: f a -> f b -> f b
(*>) = liftA2 (const id)

This caught my otherwise fragmented attention; it simply looked strange. Having reverted to using points and expanded the expression, it was soon clear that const id x = id. In other words, const id is

\pi_2 : S \times T \to T ; (s,t) \mapsto t \quad \exists \, S, T.

I would never have used const id, opting instead for a locally defined projection. In my opinion, this would have been a lot clearer, but what do I know. Any input on this? I cannot find any standard functions for projections on and injections into arguments in the standard API. Having these would enable point free definitions for a lot more functions, n'est-ce pas?