UP | HOME

My first package: eventuel
#emacs#node#gitlab

Table of Contents

I've decided to develop a special minor mode for sending events from Emacs to an external REST API and I'm calling it EVENTUEL.

I'm using Eldev for packaging and linting, NodeJS for the RESTful server, Docker for creating images, and Gitlab CI/CD for putting it all together.

Here are some of the trickier problems I've run into during the process.

1 Gitlab CI/CD

In order to package eventuel in the cloud for each commit using Eldev, I added a build stage using silex/emacs as the base image:

package:
  stage: build
  image: silex/emacs:27
  before_script:
    - curl -fsSL https://raw.github.com/doublep/eldev/master/webinstall/eldev | sh
  script:
    - /root/.eldev/bin/eldev -dtT lint
    - /root/.eldev/bin/eldev -dtT package
  artifacts:
    paths:
      - dist/

The command eldev package creates a dist folder which is uploaded to Gitlab's artifact repository. In the dist folder is a packed archive from a fresh clone that can be installed using M-x package-install-file.

2 Gitlab Release

I wanted to use the Gitlab Release api for hosting tagged versions of the package. In order to do this from the CI/CD template, I had to specify the direct url to the artifact.

release:
  stage: release
  only:
    - tags
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  dependencies:
    - package
  release:
    tag_name: $CI_COMMIT_TAG
    description: $CI_COMMIT_DESCRIPTION
    assets:
      links:
        - name: eventuel-$CI_COMMIT_TAG.tar
          url: $CI_PROJECT_URL/-/jobs/$CI_JOB_ID/artifacts/raw/dist/eventuel-$CI_COMMIT_TAG.tar
  script:
    - echo Release
  artifacts:
    paths:
      - dist/
    expire_in: never

Now, when I tag the repository (eg git tag 0.1.0 && git push origin 0.1.0), the artifacts from the package job are re-downloaded into the release job and archived once again, except this time with the flag expire_in: never to make sure Gitlab doesn't clean up this release accidentally.

The link is a direct-download link to the artifact dist/eventuel-0.1.0.tar, which will be shown on the Releases page of the repository.

Finally, the commit description (everything after the first line) is set as the release notes. I haven't figured out how to format this properly, it may be rendered as markdown.

3 Eldev file

It was tricky to figure out how to use the filesets options in Eldev, but finally I figured out how to translate the complex 'composite' fileset eldev-standard-excludes into a simple list. From there I could ignore Node depedencies and let the package install them from the web after installation.

(setq eldev-standard-excludes '(".*" "node_modules/" "/dist/"))

4 Linting dependencies and runtime dependencies

I still can't figure out how to tell npm to only install devDependencies for the linting stage in the CI/CD template, so instead I split the package.json up into two files: package.json and server/package.json. The former is only for linting while the latter is only for runtime dependencies.

5 Custom prefix for command map

I figured out a pretty elegant way to set the prefix for a global command keymap from a user-defined option.

The solution requires the easy-mmode package:

(require 'easy-mmode)

First I defined the custom prefix as nil to start:

(defcustom eventuel-prefix nil
  "Prefix key for the eventuel keymap."
  :type 'string
  :group 'eventuel)

I defined the mode map to be an empty sparse keymap:

(defvar eventuel-mode-map (make-sparse-keymap)
  "Keymap for managing nodes and users on the eventuel server.")

I defined the command map using easy-mmode-define-keymap:

(defvar eventuel-command-map
  (easy-mmode-define-keymap
   (mapcar (lambda (k) (cons (kbd (car k)) (cdr k)))
           '(("n n" . eventuel-get-nodes)
             ("n c" . eventuel-create-node)
             ("n d" . eventuel-delete-node)
             ("u u" . eventuel-get-users)
             ("u c" . eventuel-create-user)
             ("u d" . eventuel-delete-user)
             ("u p" . eventuel-set-password))))
  "Postfix keymap to be used in `eventuel-mode-map'.")

Finally I defined the minor mode with the options :keymap eventuel-mode-map. Within the startup function, I check for a user-defined prefix and modify the eventuel-mode-map. It's important not to reassign the variable or else the changes won't take effect.

(when eventuel-prefix
  (setcdr eventuel-mode-map nil) ;; Remove previous prefix bindings
  (define-key eventuel-mode-map (kbd eventuel-prefix) eventuel-command-map))

6 request.el

I made extensive use of the request library for interacting asynchronously with the REST server. I defined three utility functions which made the rest of the code a lot easier on the eyes.

6.1 eventuel–parser

This parser catches errors while parsing the JSON response and just returns the full response text instead. The server is set up so that error messages are returned in plain text.

(defun eventuel--parser ()
  "Parse responses from the eventuel server."
  (condition-case nil (json-read)
    (error
     (buffer-string))))

6.2 eventuel–error

If it were any other library than request, the error function could be a regular defun, however I had to use cl-defun in order to get the common-lisp structure needed for the callback function:

(cl-defun eventuel--error (&key error-thrown &key data &allow-other-keys)
  "Function to handle errors from the eventuel server."
  (message "Encountered an error"))

6.3 eventuel–success

Finally, I disliked the callback structure recommended by the README for request, which required a cl-function and lambda. I made a macro to simply pass elisp forms without any decoration:

(defmacro eventuel--success (&rest body)
  "This is a utility macro for the request library.

In BODY, the variable `data' will be set to the JSON response
from the server"
  (list 'cl-function
        (append (list 'lambda (list '&key 'data '&allow-other-keys))
                body)))

6.4 Example usage

(request "http://some-url"
  :type "GET"
  :parser 'eventuel--parser
  :error 'eventuel--error
  :success (eventuel--success
      (message "Succesful!")
      (pp data)))

7 NodeJS and Sqlite3

Sqlite3 is a very basic implementation of sqlite without any async functions or the tagged template literal function that I'm used to. I defined a utility function sql to do both of these things at once:

const sqlite3 = require('sqlite3');

const { DB_FILENAME } = process.env;

const db = new sqlite3.Database(DB_FILENAME);

exports.default = (strings, ...args) =>
  new Promise((resolve, reject) => {
    db.all(strings.join('?'), args, (err, rows) => {
      if (err) reject(err);
      else resolve(rows);
    });
  });

Now I can just await sql expressions without worrying about matching up args and question marks or endless nesting callback functions.

const sql = require('./sql').default;

async function getSomeStuff(isAwesome = true) {
  const stuff = await sql`
    SELECT stuff
    FROM myTable
    WHERE awesome=${+isAwesome}
  `;
  return stuff;
}

Because async functions are simply a wrapper around returning a promise, it's equivalent to simply do this:

const sql = require('./sql').default;

const getSomeStuff = (isAwesome = true) => sql`
  SELECT stuff
  FROM myTable
  WHERE awesome=${+isAwesome}
`;

8 Server sent event stream

I use an SSE stream in order to send events to the web client. On both the client and the server, the setup was a lot simpler than I thought and required no external packages. For the expressjs side, I defined a handler that takes a callback function for sending data:

exports.default = (handler) => (req, res) => {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    Connection: 'keep-alive',
  });

  handler(
    req,
    {
      send: (data) => res.write(`data: ${JSON.stringify(data)}\n\n`),
      onClose: (cb) => res.on('close', cb),
    },
  );
};

Usage:

const EventEmitter = require('events');
const { Router } = require('express');

const sse = require('./sse').default;

const logEmitter = new EventEmitter();
const router = Router();

router.get('/stream', sse((_req, conn) => {
  const listener = (entry) => conn.send(entry);
  logEmitter.on('entry', listener);
  conn.onClose(() => logEmitter.off('entry', listener));
}));

router.post('/log', (req, res) => {
  logEmitter.emit(req.body);
  return res.sendStatus(200);
});

On the client side, it's even simpler:

const logStream = new EventSource('/stream');
logStream.onmessage = (event) => {
  console.log('Received log', JSON.parse(event.data));
}

However, one bug was found for kubernetes, where my nginx ingress has a connection timeout separate and out of the control of expressjs. I had to add an annotation to the ingress in order to keep the connection open for long periods of time:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tygr-eventuel
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "86400"