LogBus

Put your logs on the bus and send them to their happy place.

Highlights

1st-Class Support for [De]Multiplexing

• Tap, fan in, and fan out easily.
• Dynamically tap to help with trouble-shooting or to temporarily increase verbosity downstream.

Scales Up

• A low-power cpu thread with 50MB of resident memory can do plenty.
• Assuming your pipeline definition does not have any messy shared state, you can run as many nodes in parallel as your sinks & sources can handle.

Scales Down

• Does not require heavy engineering & tooling to tackle many use cases.
• The standard list of plugins should be able to tackle most, if not all, of your use cases.
• A simple API keeps your plugins looking very much like simple javascript functions.

Example Use Cases

• Could be used for more than lower level log processing - think Business Process Management (BPM).
• Could be used to satisfy compliance requirements. For example, could tap into the log stream early in order to ship to secure location while other pipeline flows can filter & mutate.
• Could be a part of your observability toolkit. It is simple enough to consume metrics from other systems (see netdata example) as well as parse textual data for metrics (see web analytics example).

Compared to Others

• Often times, flow control is an afterthought which leads to configurations that look more like messy DSL's.
• With first-class de[mux] support, you are not limited to a single flow or pipeline. Without this support, you need to either a) run an overly complicated configuration, or b) run multiple simpler instances which translates to more mgmt and a waste of resources reprocessing the same data multiple times.
• LogBus is more focused on being a general purpose tool, not on being the most performant.
• The LogBus plugin story looks more like the web dev story of the 90's: drop in some simple functions that take an input (consume events) and return a value (emit events). There is no overly complicated API or SDK - how much serious engineering is up to you.

Plugins

The plugin API is simple: return an object that defines some optional methods.

• onInput(event): handle an event - typically the only method you'll need to define
• start(): typically only needed for plugins that work with data sources (eg readers, writers)
• stop(): only needed for plugins that need to clean up or close any resources (eg timers, file handles)

Here is an example "tap" plugin that lets events pass through when "on" and drops them when "off":

// Emit events based on results of `tap()`:
// - true to turn tap on
// - false to turn tap off
// Will be evaluated at start of pipeline and every `intervalSeconds`.

import {isFinite} from 'lodash'
import {LogEngine, LogEvent, PluginConfig} from '@tfks/logbus'

export default function Plugin(config: PluginConfig, logbus: LogEngine) {
  const intervalSeconds = config.intervalSeconds ? Number(config.intervalSeconds) : 60
  if (!isFinite(intervalSeconds) || intervalSeconds <= 0) {
    throw Error('invalid config: intervalSeconds must be > 0')
  }
  if (typeof config.tap !== 'function') {
    throw new Error('undefined config: tap')
  }
  const sandbox = {}
  const tap = config.tap.bind(sandbox)
  let on = false
  let timer: NodeJS.Timer

  async function start() {
    on = await tap()
    timer = setInterval(async () => { on = await tap() }, intervalSeconds * 1000)
    return {stage: logbus.stage}
  }

  function onInput(event: LogEvent) {
    if (on) logbus.event(event)
  }

  function stop() {
    if (timer) clearInterval(timer)
  }

  return {onInput, start, stop}
}