I'm playing around with electron to build a photo sharing app for fun and so this is the first time I've ever really tried using node.js from clojurescript.

The node.js api uses a lot of callbacks and so I wanted to review core async and transducers so I didn't get stuck in callback you-know-where.

But first, I thought it might be fun to try and code using only callbacks.

So here's what I came up with. Here's how I might use callbacks in clojurescript to get the contents of a directory using the node.js filesystem api.

Do you know if there's any better techniques using only callbacks? If so, please leave a comment.

Notice I chose to pass a callback (which I named cb) so that I can take control back once the callback is complete:

(defn ls-with-callback
  "List contents of a directory using callbacks"
  [root-dir cb]
  (fs.readdir 
   root-dir 
   (fn [err files] (cb files))))

For the record, anytime you need to test anything that does async calls, there's a macro named async (not just a clever name!) inside cljs.test. For example, here's how I tested ls-with-callback:

(deftest example-async-tests
  (async done
         (let [cb (fn [files] (is (not (empty? files))) (done))] 
           (fs/ls-with-callback "/" cb))))

Don't worry if it takes a while to wrap your brain around that! (It definitely took me a while) I'm very curious to dive deeper to understand exactly how that async macro works. The source code is here in case you're interested. Definitely might be the subject for another post.

But. Anyway. Where was I? Oh yeah, so as you probably know, callbacks can get unwieldy pretty quickly. For example, what if we want to filter so only directories are displayed? Well, turns out the node.js fs api has a function called stat which also requires a callback.

(defn stat-with-callback [fpath cb]
  (let [c (chan)] 
    (fs.stat fpath
             (fn [err, stat]
               (cb stat)))))

For the record: there are versions of readdir and stat that are synchronous. But I'd rather not use them in an electron app because it could make the UI appear to freeze when loading large directories.

So now, I'd like to be able to combine stat-with-callback with ls-with-callback so we can filter out directories, but look what a pain that turns out to be!

(defn list-dirs-with-callback [fpath cb]
  (let [dirs  (atom [])
        other (atom [])
        cb2 (fn [fpath fcount] 
              (fn [stat] 
                (if (.isDirectory stat) 
                  (swap! dirs conj fpath)
                  (swap! other conj fpath))
                (when (>= (+ (count @dirs) (count @other)) fcount)
                  (cb @dirs))))
        cb1 (fn [files] 
              (doseq [f files]
                (let [fpath (str (str/replace fpath #"/$" "") "/" f)]
                  (stat-with-callback fpath (cb2 fpath (count files))))))]
    (ls-with-callback fpath cb1)))

The way this works is cb1 is called when readdir is finished retrieving the contents of the directory. cb1 runs stat on each of the results and cb2 is called when stat is done gathering info about each result.

cb2 puts results into a dirs atom if they are directories, and otherwise, puts results into the other atom.

Finally, if the total number of results in dirs plus the total number of results in other is greater than or equal to the number of results found by readdir, we call the callback function supplied as an argument with the full list of directories.

Shwoo. That's the result of only 2 callbacks?! Imagine how crazy it gets with 3 or 4!

Thank goodness for clojure's core.async and transducers. That's the topic for the next post.