January 8, 2017

Packaging Js libs for Cljs

Happy New Year! I hope everyone had a great holiday. And hope you're having a good start to the new year.

Over the last week, I've been planning what to focus on for 2017 and for the next few months at least, I'd like to try learning more about the Amazon Web Services (AWS).

I've used a few of the services in the past such as EC2 and S3. There are so many interesting looking AWS services I'd like to try out.

Authentication and Authorization seems to be a good place to start. So, I'd like to explore AWS Cognito

The AWS Cognito service provides a javascript sdk here.

I'd love to be able to use the cognito library from clojurescript, but at the time of writing this, there doesn't seem to be a cljsjs package.

Also, looks like cognito js lib has several dependencies (listed here)

Of course, I could create an html page and include all the dependencies, but that is cumbersome and also doesn't take advantage of any of the code optimizations provided by the closure compiler.

In this post, I'd like to share how to package 3rd party javascript libraries into cljsjs packages so they can be easily (and efficiently) used from clojurescript.

Without further ado, here's a walk thru of how I packaged AWS Cognito SDK as a cljsjs package.

What is cljsjs?

It's basically a way to package javascript libraries so that they can easily be used inside clojurescript projects.

cljsjs uses boot which under the covers uses aether which is a java library that knows how to manage maven artifacts and repositories.

So, cljsjs packages are essentially maven artifacts.

If you aren't familiar with Maven, don't worry! No need to be a maven expert to use cljsjs. A maven artifact is really just a jar file containing code and some metadata.

Where to start

Our first step is to check to see whether any of the libraries were already available from the cljsjs project.

In order to do so, let's browse to http://cljsjs.github.io/ and search for "aws". At the time of this writing, this is the only result: [cljsjs/aws-sdk-js "2.2.41-3"].

That's the aws sdk for other services, but it doesn't include the code necessary to interact with cognito.

So, we need to create a new cljsjs package for the amazon cognito identity js libraries.

After a quick read of the github readme, It looks like there are two js files we need:

  • amazon-cognito-identity.min.js
  • aws-cognito-sdk.min.js

In addition, the README mentions that we also need the following dependencies:

  • jsbn
  • jsbn2
  • sjcl

At the time of this writing, none of these are available. So, we'll have to create a cljsjs package for each of these.

So, let's start by creating a package for sjcl, shall we?

Creating a new sjcl cljsjs package

The README from the AWS Cognito Javascript SDK Github shows this repo as the place to find the code for sjcl.

It looks like this is the actual js file we need: https://github.com/bitwiseshiftleft/sjcl/blob/master/sjcl.js

The first thing we need to do is to create a github branch. Let's name it sjcl.

git checkout master
git pull origin master
git checkout -b sjcl

When creating cljsjs packages, I always follow along with the very helpful guide here.

Let's create the directory structures for a new sjcl package:

cd cljsjs/packages
mkdir -p sjcl/resources/cljsjs/sjcl/common

Create Externs

An extern file is sort of like an interface, or public api definition for a javascript library which is used by the Google Closure Compiler in order to minify and optimize javascript code. You can read more about creating extern files here.

You can create extern files by hand. There are also automatic generators available. For this example, we'll use the automatic extern generator found here.

To generate the externs, simply enter the url to the sjcl library and then enter sjcl as the js object to extern.

Github provides urls like this one to get the "raw" contents of a file:

https://raw.githubusercontent.com/bitwiseshiftleft/sjcl/master/sjcl.js

But unfortunately, that won't work in the externs generator. It has to do with the MIME type used by github to serve that file. For a workaround, you can replace raw.githubusercontent.com with rawgit.com and that should solve it.

So, let's try entering https://raw.git.com/bitwiseshiftleft/sjcl/master/sjcl.js and sjcl into the externs generate.

And then copy the results into cljsjs/packages/sjcl/resources/cljsjs/sjcl/common/sjcl.ext.js.

sjcl build.boot

Ok, next up is the build.boot file.

Let's make a copy of cljsjs/packages/aws-sdk-js/build.boot. Edit the build.boot and change aws-sdk-js to sjcl.

It looks like the current version of sjcl is 1.0.6, so update the version to match as well.

The next step is to write the package task. This is the most involved part of creating cljsjs packages.

Conceptually, we need to download the js file, and move it to the appropriate directories so that the boot package task can create a jar.

Here's what I came up with:

(deftask package []
  (comp
   (download :url (format (str "https://raw.githubusercontent.com/"
                               "bitwiseshiftleft/sjcl/%s/sjcl.js") 
                          +lib-version+)
             :checksum "??"
             :unzip false)

   (sift :move {#"^sjcl.js" "cljsjs/sjcl/development/sjcl.inc.js"
                #"^sjcl.js" "cljsjs/sjcl/production/sjcl.min.inc.js"})

   (sift :include #{#"^cljsjs"})

   (deps-cljs :name "cljsjs.sjcl")
   (pom)
   (jar)))

Let's run boot package and see what happens!

cd cljsjs/packages/sjcl
boot package

Downloading sjcl.js
     clojure.lang.ExceptionInfo: Checksum of file sjcl.js in not ?? but D7DC40AD6718245B6B5F158F621FC4E4

Oops, got an error, but it was expected. Since I didn't know the checksum of the sjcl.js file, I entered "??" above. Now we can fix that by coping the correct checksum (D7DC40AD6718245B6B5F158F621FC4E4) as shown in the error message.

So, update the checksum and try again.

Downloading sjcl.js
Sifting output files...
Sifting output files...
Writing deps.cljs
Writing pom.xml and pom.properties...
Writing sjcl-1.0.6-1.jar...

Awesome! We can commit this and submit pull request to cljsjs/packages project in order to get it into clojars. But in the meantime, don't forget to run boot package install target.

Once the maintainers of the cljsjs/packages project accept our pull request and the update makes it to clojars, then there's no need to install locally (because boot and the underlying maven machinery will automatically discover the artifact in clojars).

But for now, until it's available on clojars, we need to run the boot install command in order to use this new package.

In order to submit a pull request, remember to use the cljsjs guidelines for creating commit comments. For example, here's the git commands I used to push these new changes:

git commit -am "[sjcl] New package"
git push upgradingdave sjcl

Creating the jsbn and jsbn2 cljsjs package

Creating this next package is almost exactly the same as creating the sjcl package. It's almost not even worth mentioning, except for one interesting difference.

Since there are two js files that are very closely related, it makes sense to create a cljsjs package that contains 2 namespaces:

  • cljsjs.jsbn
  • cljsjs.jsbn2

So, I created a new branch, and created the extern file just the same as before.

The build.boot file is almost exactly the same as well, except notice that instead of using the cljs-deps task, I included a custom deps.cljs file:

(deftask package []
  (comp
   (download :url "http://www-cs-students.stanford.edu/~tjw/jsbn/jsbn.js"
             :checksum "5E6FE0DA9EF45687E52781B9A646454E"
             :unzip false)

   (download :url "http://www-cs-students.stanford.edu/~tjw/jsbn/jsbn2.js"
             :checksum "402CB831E9747F7B0029CE06B8CD76BD"
             :unzip false)

   (sift :move {#"^jsbn.js" "cljsjs/jsbn/development/jsbn.inc.js"
                #"^jsbn.js" "cljsjs/jsbn/production/jsbn.min.inc.js"

                #"^jsbn2.js" "cljsjs/jsbn/development/jsbn2.inc.js"
                #"^jsbn2.js" "cljsjs/jsbn/production/jsbn2.min.inc.js"})

   (sift :include #{#"^cljsjs" #"^deps\.cljs"})

   (pom)
   (jar)))

Here's the handmade deps file I created:

{:foreign-libs 
 [{:file "cljsjs/jsbn/development/jsbn.inc.js", 
   :provides ["cljsjs.jsbn"]}
  {:file "cljsjs/jsbn/development/jsbn2.inc.js", 
   :provides ["cljsjs.jsbn2"]
   :requires ["cljsjs.jsbn"]}], 
 :externs ["cljsjs/jsbn/common/jsbn.ext.js"]}

I saved that into jsbn/resources/deps.cljs.

After creating the build.boot and custom deps.cljs, I was able to run boot package install target successfully.

2 packages down ... 1 more to go.

Creating the aws-cognito-identity-js package

I used pretty much the exact same steps to create this last package. This is another package that includes 2 js files (and 2 namespaces so I'll hand write the deps.cljs):

  • amazon-cognito-identity-js.inc.js
  • aws-cognito-sdk.js

Here's the packages task from the build.boot file:

(deftask package []
  (comp
   (download :url (format "https://github.com/aws/amazon-cognito-identity-js/archive/v%s.zip" +lib-version+)
             :checksum "E3D5E3330ED8B7140913A69351E457FC"
             :unzip true)

   (sift :move {#"^amazon-cognito-identity-js-.*/dist/amazon-cognito-identity.min.js" 
                "cljsjs/aws-cognito-identity-js/development/amazon-cognito-identity-js.inc.js"

                #"^amazon-cognito-identity-js-.*/dist/amazon-cognito-identity.min.js" 
                "cljsjs/aws-cognito-identity-js/production/amazon-cognito-identity-js.min.inc.js"

                #"^amazon-cognito-identity-js-.*/dist/aws-cognito-sdk.js" 
                "cljsjs/aws-cognito-identity-js/development/aws-cognito-sdk.inc.js"

                #"^amazon-cognito-identity-js-.*/dist/aws-cognito-sdk.min.js" 
                "cljsjs/aws-cognito-identity-js/development/aws-cognito-sdk.min.inc.js"})

   (sift :include #{#"^cljsjs" #"^deps\.cljs"})

   (pom)
   (jar)))

Here's the deps.cljs. Notice the requires keys show the dependencies of each library:

{:foreign-libs 
 [{:file "cljsjs/aws-cognito-identity-js/development/aws-cognito-sdk.inc.js", 
   :provides ["cljsjs.aws-cognito-sdk"]}
  {:file "cljsjs/aws-cognito-identity-js/development/amazon-cognito-identity-js.inc.js", 
   :provides ["cljsjs.aws-cognito-identity-js"]
   :requires ["cljsjs.aws-cognito-sdk"
              "cljsjs.sjcl"
              "cljsjs.jsbn"]}],
 :externs ["cljsjs/aws-cognito-identity-js/common/aws-cognito-identity-js.ext.js"]}

Summary

Done! We created 3 cljsjs packages, containing a total of 5 js library files.

And now we can authenticate against cognito from clojurescript!

In order to actually use this new package, I added the 3 dependencies like this:

[cljsjs/aws-cognito-identity-js "1.11.0-1"]
[cljsjs/jsbn                    "0.1.0-1"]
[cljsjs/sjcl                    "1.0.6-1"]

And then I can call the AWS Cognito service api, for example, like this:

(require '[cljsjs.aws-cognito-identity-js :as cognito])

(defn create-cognito-user-pool [user-pool]
  (js/AWSCognito.CognitoIdentityServiceProvider.CognitoUserPool. user-pool))

(defn signup [user-pool username password attribute-list]
  (.signUp user-pool username password (clj->js attribute-list) nil signup-cb))

Hope that gives a good overview of what's involved to create a new cljsjs package.

I'm really liking this cognito service so far. Hopefully it will save me a lot of user management related work from here on out!

Tags: clojure clojurescript