My first package: eventuel
#emacs#node#gitlab
Table of Contents
I've decided to develop a special minor mode for sending events from Emacs to an external REST API and I'm calling it EVENTUEL.
I'm using Eldev for packaging and linting, NodeJS for the RESTful server, Docker for creating images, and Gitlab CI/CD for putting it all together.
Here are some of the trickier problems I've run into during the process.
1. Gitlab CI/CD
In order to package eventuel in the cloud for each commit using
Eldev, I added a build
stage using silex/emacs
as the base
image:
package: stage: build image: silex/emacs:27 before_script: - curl -fsSL https://raw.github.com/doublep/eldev/master/webinstall/eldev | sh script: - /root/.eldev/bin/eldev -dtT lint - /root/.eldev/bin/eldev -dtT package artifacts: paths: - dist/
The command eldev package
creates a dist
folder which is
uploaded to Gitlab's artifact repository. In the dist folder is a
packed archive from a fresh clone that can be installed using
M-x package-install-file
.
2. Gitlab Release
I wanted to use the Gitlab Release api for hosting tagged versions of the package. In order to do this from the CI/CD template, I had to specify the direct url to the artifact.
release: stage: release only: - tags image: registry.gitlab.com/gitlab-org/release-cli:latest dependencies: - package release: tag_name: $CI_COMMIT_TAG description: $CI_COMMIT_DESCRIPTION assets: links: - name: eventuel-$CI_COMMIT_TAG.tar url: $CI_PROJECT_URL/-/jobs/$CI_JOB_ID/artifacts/raw/dist/eventuel-$CI_COMMIT_TAG.tar script: - echo Release artifacts: paths: - dist/ expire_in: never
Now, when I tag the repository (eg git tag 0.1.0 && git push origin
0.1.0
), the artifacts from the package
job are re-downloaded into
the release
job and archived once again, except this time with the
flag expire_in: never
to make sure Gitlab doesn't clean up this
release accidentally.
The link is a direct-download link to the artifact
dist/eventuel-0.1.0.tar
, which will be shown on the Releases
page of the repository.
Finally, the commit description (everything after the first line) is set as the release notes. I haven't figured out how to format this properly, it may be rendered as markdown.
3. Eldev file
It was tricky to figure out how to use the filesets options in
Eldev, but finally I figured out how to translate the complex
'composite' fileset eldev-standard-excludes
into a simple
list. From there I could ignore Node depedencies and let the package
install them from the web after installation.
(setq eldev-standard-excludes '(".*" "node_modules/" "/dist/"))
4. Linting dependencies and runtime dependencies
I still can't figure out how to tell npm
to only install
devDependencies
for the linting stage in the CI/CD template, so
instead I split the package.json
up into two files:
package.json
and server/package.json
. The former is only for
linting while the latter is only for runtime dependencies.
5. Custom prefix for command map
I figured out a pretty elegant way to set the prefix for a global command keymap from a user-defined option.
The solution requires the easy-mmode package:
(require 'easy-mmode)
First I defined the custom prefix as nil to start:
(defcustom eventuel-prefix nil "Prefix key for the eventuel keymap." :type 'string :group 'eventuel)
I defined the mode map to be an empty sparse keymap:
(defvar eventuel-mode-map (make-sparse-keymap) "Keymap for managing nodes and users on the eventuel server.")
I defined the command map using easy-mmode-define-keymap
:
(defvar eventuel-command-map (easy-mmode-define-keymap (mapcar (lambda (k) (cons (kbd (car k)) (cdr k))) '(("n n" . eventuel-get-nodes) ("n c" . eventuel-create-node) ("n d" . eventuel-delete-node) ("u u" . eventuel-get-users) ("u c" . eventuel-create-user) ("u d" . eventuel-delete-user) ("u p" . eventuel-set-password)))) "Postfix keymap to be used in `eventuel-mode-map'.")
Finally I defined the minor mode with the options :keymap
eventuel-mode-map
. Within the startup function, I check for a
user-defined prefix and modify the eventuel-mode-map
. It's
important not to reassign the variable or else the changes won't
take effect.
(when eventuel-prefix (setcdr eventuel-mode-map nil) ;; Remove previous prefix bindings (define-key eventuel-mode-map (kbd eventuel-prefix) eventuel-command-map))
6. request.el
I made extensive use of the request
library for interacting
asynchronously with the REST server. I defined three utility
functions which made the rest of the code a lot easier on the eyes.
6.1. eventuel–parser
This parser catches errors while parsing the JSON response and just returns the full response text instead. The server is set up so that error messages are returned in plain text.
(defun eventuel--parser () "Parse responses from the eventuel server." (condition-case nil (json-read) (error (buffer-string))))
6.2. eventuel–error
If it were any other library than request, the error function
could be a regular defun
, however I had to use cl-defun
in order
to get the common-lisp structure needed for the callback function:
(cl-defun eventuel--error (&key error-thrown &key data &allow-other-keys) "Function to handle errors from the eventuel server." (message "Encountered an error"))
6.3. eventuel–success
Finally, I disliked the callback structure recommended by the
README for request, which required a cl-function
and lambda
. I
made a macro to simply pass elisp forms without any decoration:
(defmacro eventuel--success (&rest body) "This is a utility macro for the request library. In BODY, the variable `data' will be set to the JSON response from the server" (list 'cl-function (append (list 'lambda (list '&key 'data '&allow-other-keys)) body)))
6.4. Example usage
(request "http://some-url" :type "GET" :parser 'eventuel--parser :error 'eventuel--error :success (eventuel--success (message "Succesful!") (pp data)))
7. NodeJS and Sqlite3
Sqlite3 is a very basic implementation of sqlite without any async
functions or the tagged template literal function that I'm used
to. I defined a utility function sql
to do both of these things at
once:
const sqlite3 = require('sqlite3'); const { DB_FILENAME } = process.env; const db = new sqlite3.Database(DB_FILENAME); exports.default = (strings, ...args) => new Promise((resolve, reject) => { db.all(strings.join('?'), args, (err, rows) => { if (err) reject(err); else resolve(rows); }); });
Now I can just await sql expressions without worrying about matching up args and question marks or endless nesting callback functions.
const sql = require('./sql').default; async function getSomeStuff(isAwesome = true) { const stuff = await sql` SELECT stuff FROM myTable WHERE awesome=${+isAwesome} `; return stuff; }
Because async functions are simply a wrapper around returning a promise, it's equivalent to simply do this:
const sql = require('./sql').default; const getSomeStuff = (isAwesome = true) => sql` SELECT stuff FROM myTable WHERE awesome=${+isAwesome} `;
8. Server sent event stream
I use an SSE stream in order to send events to the web client. On both the client and the server, the setup was a lot simpler than I thought and required no external packages. For the expressjs side, I defined a handler that takes a callback function for sending data:
exports.default = (handler) => (req, res) => { res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', Connection: 'keep-alive', }); handler( req, { send: (data) => res.write(`data: ${JSON.stringify(data)}\n\n`), onClose: (cb) => res.on('close', cb), }, ); };
Usage:
const EventEmitter = require('events'); const { Router } = require('express'); const sse = require('./sse').default; const logEmitter = new EventEmitter(); const router = Router(); router.get('/stream', sse((_req, conn) => { const listener = (entry) => conn.send(entry); logEmitter.on('entry', listener); conn.onClose(() => logEmitter.off('entry', listener)); })); router.post('/log', (req, res) => { logEmitter.emit(req.body); return res.sendStatus(200); });
On the client side, it's even simpler:
const logStream = new EventSource('/stream'); logStream.onmessage = (event) => { console.log('Received log', JSON.parse(event.data)); }
However, one bug was found for kubernetes, where my nginx ingress has a connection timeout separate and out of the control of expressjs. I had to add an annotation to the ingress in order to keep the connection open for long periods of time:
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: tygr-eventuel annotations: nginx.ingress.kubernetes.io/proxy-read-timeout: "86400"