QFiltr

QFiltr is a powerful but simple filter, limiter & queueing system that can be used to moderate or maintain a data stream. It was developed as part of a M2M/IoT cloud project in order to prevent devices from reporting data at too high rate, but has since been adapted as a standalone module for many possible uses.

Some use cases:

• Use within an API in order to rate limit incoming requests.
• Use with UI interactions, like a button that can be rapidly pressed, to run a function via queue or rate limit.
• Use on a chat server, to detect and block spam.
• Use with M2M/IoT Cloud Data to limit or queue devices that report too rapidly.

Installation

QFiltr can be installed for use in a Node.js application via NPM, or for use in a basic javascript application or website by getting the standalone qfiltr.js from the GitHub Repository.

NPM

npm install qfiltr

Javascript

Get the standalone version from https://github.com/msudol/qfiltr

• Standalone version is in /tests/qfiltr.js
• See how to embed in javascript in /tests/basic-test.html

Basic Usage

Require the qfilter module and create a new instance of qfiltr.

var qfiltr = require("qfiltr");
var qFiltr = new qfiltr();  

qFiltr.COMMAND("id", {options}, callback(s))

Functions

qfiltr.limit(“id”, {limitCount:3, limitTime:1000}, success, fail)

Limit is a basic hard limit, where anything over X messages in Y seconds returns the fail callback, while anything that is under the limit returns the success callback.

• Options Object:
  • limitCount:3
  • limitTime:1000
• Callbacks:
  • Allow (Success)
  • Block (Fail)

Example: No more than 5 messages or commands in 500ms.

qFiltr.limit("limitExample", {limitCount:5, limitTime:500}, function() {
    console.log("Allowed message");
}, function() { 
    console.log("Blocked message");
});

qfiltr.queue:

Queue is basic queueing function, feed messages in at any rate, and they are processed at the queue settings rate until the queue runs out.

• Options Object:
  • queueTimer:1000
  • queueMax:-1 (no limit)
• Callbacks:
  • OnQueue (Success)
  • Queue Ended
  • Maxed (Queue is full)

Example: Queue incoming messages or commands to be run every 2000ms until they are all run

var qfiltr = require("qfiltr");
qFiltr = new qfiltr();

// call a bunch of messages really fast
for (var i = 0; i < 10; i++) {
    sendMessage("This is message " + i);
}

function sendMessage(message) {
    // add message to qFilter.queue with callback functions
    qFiltr.queue("msgQ", {queueTimer:2000}, function() {
        console.log(message);
    }, function() {
        console.log("Queue complete");
    });  
}

qfiltr.qlimit:

QLimit is combo function combining rate limiting and queueing function, feed messages in at any rate until they exceed the rate limit, and then they are processed at the queue settings rate until the queue runs out.

• Options Object:
  • limitCount:3
  • limitTime:1000
  • queueTimer:1000
  • queueMax:-1 (no limit)
• Callbacks:
  • Allow (Success)
  • LimitReached (Queue Starting)
  • Queue Ended
  • Maxed (Queue is full)

Example: Send 100 messages at random intervals, rate limit if it goes too fast and put into queue mode

var qfiltr = require("qfiltr")
qFiltr = new qfiltr();

var sendStop = 0
var overRate = false;
var testAdjuster = 0;
 
// send messages at random intervals
function sender(t) { 
    if (sendStop > 100) return;
    sendStop++;
    
    if (overRate) { testAdjuster = 20 }
    else { testAdjuster = 0}
    
    setTimeout(function() {
        // generate new t
        var i = Math.floor(Math.exp(Math.random()*Math.log(51 + testAdjuster)));
        sendMessage("Message #" + sendStop + " sent at " + i*10 + "ms");
        // run again
        sender(i*10);
    }, t);
}
 
function sendMessage(message) {
    // add message to qFilter.queue with callback functions 
    qFiltr.qlimit("msgQ", {limitCount:10, limitTime:1000, queueTimer:100}, function() {
        console.log(message);
    }, function() {
        overRate = true;
        console.log("Rate limit exceded, queuing data");
    }, function() {
        overRate = false;
        console.log("Queue cleared resuming normal operation");        
    });  
}

sender(1000);

qfilter.filter:

The filter function is in development, and it’s design will be tailored more toward something like a chat server, in which a condition will need to be met in order for the message to be allowed.

Objects

The ID that you set in a filter function writes an entry into a “datastore”, so that you can have multiple filters running with different settings.

This ID will also allow you to check if the current ID’s queue is running or not.

Accessing the datastore

var idStore = qFilter.dataStore[ID];
console.log("Items current in queue:" + idStore.length);

Get Queue State

var isQRunning = qFilter.qRunning[ID];

TODO

• qfiltr.filter: pass filter regex / matching conditions along with message