Dashflow —automate local development tasks

The Local Development Challenge

Developers used to live in a simple world with simple enough developer experiences.

Edit a file, run it, and test it. That’s it.

Unfortunately, that is no longer the case today. As a result of diverse technologies and software systems continuing to grow in complexity, the developer experience has become more and more fragmented.

To answer the question of how fragmented it has been already, let’s follow a day in the life of a web developer, named Mr. James Bond.

Mr. James Bond’s job involves working on a web application, which is architected in the popular Single Page Application + Microservices design style. The frontend is built in React and the backend is split into 3 services: user-service, order-service, and BFF-service (Backend For Frontend).

In order to start development locally, Mr. Bond starts 4 terminal windows, each running a process. Whenever he changes a few lines of code in his IDE, he’ll switch back to one of the terminal windows, quit the running process and start it again. If Mr. Bond would like to run unit tests for the frontend app, he would have to open the fifth terminal window and trigger unit tests each time after he touches the code. At the end of the day, unsurprisingly, Mr. Bond ended up opening over 10 terminal windows. Even finding the right window for the right process became a challenge for him.

Mr. Bond investigated solutions such as using docker-compose to make his life easier. It does help him to start those services at once. But it doesn’t help too much when it comes to dev tasks such as running unit test, linting, etc.

A simple rule of thumb is, the more components we have in a system, the more rules developers have to memorize to keep compliance: the lint must be passing, the test must be passing, etc, these are mental overheads.

We’ve found it painful to memorize those workflow rules and repeat all dev tasks manually. Is there a good way to automate those tasks, yet still allow us to retrieve the information we would like to see whenever necessary? We did some investigation but no single solution solved our pain points perfectly. So we decided to build one, which we called “Dashflow.”

How Dashflow Solves This

Simply put, Dashflow is a command line job runner. It takes a YAML configuration file which describes what processes we would like to run in the background and what workflow rules to enforce whenever certain events happen. This is a sample YAML config file:

commands:
  lint:
    shell: { cmd: yarn --silent lint }
  test:
    shell: { cmd: yarn --silent test --no-color --verbose }

streams:
  server:
    shell: { cmd: yarn start }
  watch-src:
    watch: { glob: "src/**/*.js" }
  watch-test:
    watch: { glob: "test/**/*.js" }

workflows:
  init:
    match: SYSTEM:started
    parallel:
      - command: lint
      - command: test
  restart-server:
    match: watch-src:.*
    restart: server
  src-lint-then-test:
    match: watch-src:.*
    serial:
      - command: lint
      - command: test
  test-lint-and-test:
    match: watch-test:.*
    parallel:
      - command: lint
      - command: test

dashboards:
  all-in-one:
    - log:
        position: 0 0 100% 50%
        title: Server
        filter: stream:server:.*
    - log:
        position: 0 50% 50% 100%
        title: Lint
        filter: command:lint:.*
    - log:
        position: 50% 50% 100% 100%
        title: Test
        filter: command:test:.*
  lint-only:
    - log:
        position: 0 0 100% 100%
        title: Lint
        filter: command:lint:.*

The careful readers may have noticed that we have a commands section in the YAML file. That’s a useful feature! You can use dashflow as your simple Makefile:

dashflow lint
dashflow test

What is unique to Dashflow is the daemon mode, when you run the dashflow command without any arguments, in a folder that contains dashflow.yml.

When Dashflow starts, it runs all described processes (streams section) in the background, tracking their outputs as “streams”. All stream outputs are normalized as events. The workflow engine listens to events and triggers additional “commands” when certain events happen, which generates subsequent events.

All events are tracked in memory and disappear if the Dashflow process is restarted.

To make it easy to explore events, Dashflow has a built-in web server that serves a web page where you can view “dashboards.” Those dashboards are defined in the same YAML file, in the dashboards section. With the screenshot below we can see how the definitions map to the actual layout.

web dashboard

Dashflow also comes with an interactive shell which can be used to attach to running processes, or to inspect items in the event queue.

interactive shell

With Dashflow, we only have to define our development workflow once, and never have to worry about those repetitive tasks again. The configuration is represented as a YAML file, which can be stored in the code repo, and shared to all members of the team. Team members just need to install Dashflow and they can start local development without any configurations!

At FreeWheel, a Comcast Company, we’re happy with local development again, all thanks to the help of Dashflow. We hope that by open sourcing Dashflow, more developers and teams can be freed from the fragmented local development experience and be happy building software again.

---

Originally published on Medium on June 18, 2019.