OpenAF processor filters

The OpenAF processor filters are applied in the following order:

input data -> path=… -> apply transformers to data -> from=… -> sql=… -> output data

---

For more advanced cases they can also be applied in the following order:

input data -> ifrom=… -> isql=… -> path=… -> apply transformers to data -> from=… -> sql=… -> opath=… -> output data

🪚 Path

The path= filter tool is based on the JMESPath library. To see all the available options please refer to http://jmespath.org. Here are some examples:

Slicing

When querying arrays it’s possible to “slice” them in different ways (examples: the first five; from the third to the fifth; the first two; the last element; etc.)

• “[0:5]”
• “[5:10]”
• ”[:5]”
• ”[::2]”
• ”[::-1]”

Projections

As with projections it’s also possible to “project” multiple fields.

• “a[*].first”
• “a.*.b”
• ”[]”

Filters

It’s possible to apply simple conditions (including using functions) to query an array.

• “a[?b==’xyz’].c”
• “a[?b>1].x”

MultiSelect

As with projections it’s also possible to “project” multiple fields.

• “a[].[x, y]”
• “a[].{ x: x, y: y }”

Pipe

Using the same unix “pipe” mechanism it’s possible to apply different categories of queries in sequence.

• “a[*].b

  [0]”

Functions

Function	OpenAF	Description	Example
a2m(arrFields, arrValues)	20240209	Builds a map given an array of fields and values	a2m(k, v)
a4m(arr, ‘key’, dontRemove)	20240209	Converts an array into a map	a4m([], ‘name’, `true`)
abs(number)	base	The absolute value of a numeric field	[].{y:abs(y)}
add(number, number)	20240217	Adds two numbers	add(1, 2)
amerge(x, y)	20240215	OpenAF’s merge with support for arrays	amerge([], from_slon(‘(stamp:123)’))
avg(arrayNumber)	base	The average value of an array of numeric fields	avg([].y)
ceil(number)	base	Returns the smallest integer that is equal or less than a specific numeric field value	[].ceil(y)
concat(x, y)	20240209	Concats arrays or strings	concat(‘abc’, ‘123’)
contains(string/array, any)	base	Returns true of false if a string field contains a specific value	files[?contains(filename, ‘openaf.jar’) == true
count_by(arr, ‘field’)	all	Returns a count by array with the ‘_count’ per value of the provided ‘field’	count_by([], ‘isFile’)
date_diff(field, ‘unit’, nullval)	20240228	Given a date field will return the numeric difference to now for the provided unit (e.g. seconds, minutes, hours, days, months, weeks, years). Optionally a nullval will be used if no difference can be calculated	date_diff(modifiedDate, ‘days’, ‘-1’)
delete(map, ‘field’)	all	Remove a field from the provided map	delete(data, ‘timestamp’)
div(a, b)	20240217	Divides two numbers.	div(6, 2)
ends_with(string, array)	base	Returns true if a field has the provided suffix	files[?ends_with(filename, ‘.jar’)]
flat_map(x)	all	Returns a flat, one level, version of the provided map.	flat_map(data)
floor(number)	base	Returns the greatest integer that is equal or greater than a specific numeric field value	[].floor(y)
format(x, ‘format’)	20240209	OpenAF’s function $f (similar to printf) with type conversion	format(to_number(123.12), ‘%05.0f’)
formatn(x, ‘format’)	20240209	OpenAF’s function $ft (similar to printf) without type conversion	format(string, ‘%10s’)
from_bytesAbbr(x)	20240209	Given a string with a byte abbreviation will convert to bytes	from_bytesAbbr(‘12GB’)
from_datef(date, ‘format’)	20240228	Converts a date type into a string given a ‘format’ (equivalent to OpenAF’s ow.format.fromDate)	from_datef(to_datef(‘20240202’, ‘yyyyMMdd’), ‘yyyy’)
from_json(str)	20240215	Converts a json string representation into an object	from_json(‘{a:123}’)”
from_ms(x, ‘format’)	20240209	Shortcut for OpenAF’s ow.format.elapsedTime4ms function. The format is represented as a SLON/JSON string	from_ms(12000,’(abrev:true)’)
from_siAbbr(x)	20240209	Given a string with SI numeric abbreviation will convert it to the absolute value	from_siAbbr(‘100m’)
from_slon(obj)	20240215	Converts a slon string representation into an object	from_slon(‘(abc: 123)’)
from_timeAbbr(x)	20240209	Converts a time abbreviation into ms	from_timeAbbr(‘12s’)
get(nameOrPath)	20240305	Given a path to the original object or a name set by ‘set’ or ‘setp’ returns the corresponding value	packages[].{name: name, version: version, parentVersion: get(‘version’) }
group(arr, ‘field’)	all	Given an array will return a new array grouping the entries for each value of the provided field	group(files, ‘isDirectory’)
group_by(arr, ‘field1,field2’)	all	Given ar array will return a multi-level array grouping entries for each value of the provided fields (comma delimited)	group_by(files, ‘isFile, permissions’)
index_of(str, ‘search’)	20240209	Given a string will return the first index where the word ‘search’ occurs	index_of(‘This is a test’, ‘test’)
insert(obj, ‘field’, value)	20240302	Adds a ‘field’ with the corresponding value to the object.
join(string, arrayString)	base	Returns a delimited list with the values of a specific array field	join(‘, ‘, files[].filename)
keys(object)	base	Returns a list of fields for a corresponding map	keys(files[0])
last_index_of(str, ‘search’)	20240209	Given a string will return the last index where the word ‘search’ occurs	last_index_of(‘Test of a test’, ‘test’)
length(string/array/object)	base	Returns the size of any array or list	length(keys(files[0]))
lower_case(str)	20240209	Given a string returns the lowercase converted version	lower_case(‘AbC’)
m2a(arrFields, obj)	20240209	Given an array of fields returns an array with the corresponding values of the provided object map.	m2a(from_slon(‘[name	size]’), @)
m4a(obj, ‘key’)	20240209	Given an object map where each entry corresponds to a map will return an array of maps with the extra field ‘key’ representing the corresponding map entry.	m4a(obj, ‘key’)
map(expression, array)	base	Returns an array mapping	map(&filename == ‘openaf.jar’, files[])
match(str, ‘re’, ‘flags’)	20240209	Equivalent to javascript’s match function return a boolean value if a regular expression is matched to a string with optional flags	match(‘abc 123’, ‘\d+’, ‘g’)
max(number)	base	Returns the maximum of a numeric field	max(files[].size)
max_by(array, expression)	base	Returns the element for which the expression is the maximum	max_by(files[], &size)
merge(object, object)	base	Returns the merge of two objects	merge([0],[1])
min(number)	base	Returns the minimum of a numeric field	min(files[].size)
min_by(array, expression)	base	Returns the element for which the expression is the minimum	min_by(files[], &size)
mod(a, b)	20240217	Returns the modular of two numbers	mod(5, 2)
mul(a, b)	20240217	Multiplies two numbers	mul(5, 2)
not_null(any)	base	Returns the non-null value between the provided fields	[].not_null(a,b)
now(diff)	20240302	Returns the current unix timestamp number with a negative diff (or positive for dates in the future)
nvl(field, value)	20240216	Returns the provided value in case a field value is undefined or null	nvl(nullField, ‘n/a’)
replace(str, ‘re’, ‘flags’, ‘replaceText’)	20240209	Equivalent to Javascript’s replace function that given a string will search for a regular expression, with the optional flags, a replace with the provided text	replace(‘This is a test’, ‘ a’, ‘i’, ‘ not a’)
reverse(array)	base	Reverse the provided array	“reverse(@)”
search_keys(arr, ‘text’)	all	Returns an array of entries where ‘text’ was found as part of an object property.	search_keys(files, ‘filename’)
search_values(arr, ‘text’)	all	Returns an array of entries where ‘text’ was found as part of an object property value.	search_values(files, ‘.git’)
set(obj, path)	20240305	Given an object will set the value to a name that it’s equal to it’s path	set(@, ‘version’).packages[].{ name: name, parentVersion: get(‘version’) }
setp(obj, path, name)	20240305	Given an object will set the value to a name given the provided path	set(@, ‘version’, ‘ver’).packages[].{ name: name, parentVersion: get(‘ver’) }
sort(array)	base	Sorts the provided array	“sort(@)”
sort_by(array, expression)	base	Sorts the provided array by the provided expression	sort_by(files[], &size)
split(str, ‘sep’)	20240209	Equivalent to the split Javascript’s function for a string given a separator	split(@, ‘\n’)
split_re(str, ‘re’)	20240228	Equivalent to the split Javascript’s function for a string given a regular expression separator	split_re(@, ‘\s+’)
split_sep(str, sep, arrEnc)	20240217	Given a string, a separator regexp and an array of pairs of enclosure chars	split_sep(@, ‘\s+’, from_slon(‘[[’{‘	’}’]	[’(‘	’)’]]’))
starts_with(string, array)	base	Returns true if a field has the provided prefix	files[?starts_with(filename, ‘openaf.jar’)]
sub(a, b)	20240217	Substracts two numbers	sub(2, 2)
substring(str, ini, end)	all	Given a string will return a sub-string starting on the initial index until the ending index	substring(@, index_of(‘test’), 5)
sum(array)	base	Sums the numberic field of a provided array	sum(files[].size)
t(obj, ‘template’)	20240228	Applies the Handlebars ‘template’ to the provided array or map	t(@, ‘ ()’)
tF(obj, ‘template’)	20240228	Applies the Handlebars ‘template’, with all OpenAF helpers, to the provided array or map	tF(@, ‘: ‘)
template(a, ‘template’)	20240209	Applies the Handlebars ‘template’ to the provided array or map	t(@, ‘ ()’)
templateF(x, ‘template’)	20240209	Applies the Handlebars ‘template’, with all OpenAF helpers, to the provided array or map	tF(@, ‘: ‘)
timeago(num)	20240209	Given a ms timestamp will return a human readable string of how log ago that timestamp occurred.	files[].{path:filepath,howLongAgo:timeago(lastModified)}
to_array(any)	base	Transforms any input into an array	to_array(true)
to_bytesAbbr(x)	20240209	Given an absolute number of bytes will return a string with unit abbreviation.	to_bytesAbbr(12345678)
to_date(x)	20240209	Tries to convert a value to a date	to_date(createDate)
to_datef(str, ‘pattern’)	20240228	Uses a Java date format to convert a string into a date	to_datef(createDate, ‘yyyyMMdd’)
to_isoDate(x)	20240209	Tries to convert a string into an ISO date format string	to_isoDate( to_datef(createDate, ‘yyyyMMdd’) )
to_json(obj, ‘space’)	20240215	Given an object will return the JSON string representation of it with the provided spacing	to_json(@, ‘’)
to_map(arr, ‘field’)	all	Given an array it will return a map where each entry is a property using the provided field with a map as value.	to_map(files, ‘filename’)
to_numAbbr(num)	20240209	Given an absolute number will return a string with SI abbreviation	to_numAbbr(12345678)
to_number(any)	base	Transforms any input into a number	to_number(123)
to_slon(obj)	20240215	Given an object will return the SLON representation of it.	to_slon(@)
to_string(any)	base	Transforms any input into a string	to_string(123)
trim(str)	20240209	Given a string will return a trimmed version of it	trim(@)
type(any)	base	Returns the type of any input	type(to_number(123))
unique(arr)	all	Given an array will return a new de-duplicated array.	unique([])
upper_case(str)	20240209	Given a string returns the uppercase converted version	upper_case(‘AbC’)
values(a)	base	Returns an array with all the values of a map	values(files[0])

Example:

# Given all AWS EC2 instances in an account produces a table with name, type, vpc and private ip sorted by vpn

aws ec2 describe-instances | ./oafp path="Reservations[].Instances[].{name:join('',Tags[?Key=='Name'].Value),type:InstanceType,vpc:VpcId,ip:PrivateIpAddress} | sort_by(@, &vpc)" output=ctable

🏹 From

Function
all
and
andBegin
andBetween
andBetweenEquals
andContains
andEmpty
andEnds
andEquals
andGreater
andGreaterEquals
andIs
andLess
andLessEquals
andMatch
andNot
andNotBetween
andNotBetweenEquals
andNotContains
andNotEmpty
andNotEnds
andNotEquals
andNotGreater
andNotGreaterEquals
andNotIs
andNotLess
andNotLessEquals
andNotMatch
andNotStarts
andNotType
andNotWhere
andStarts
andType
andWhere
any
apply
assign
at
attach
average
averageBy
begin
between
betweenEquals
cartesian
contains
count
countBy
define
detach
distinct
each
empty
end
ends
equals
except
filter
first
fnBy
greater
greaterEquals
group
groupBy
head
ignoreCase
intersect
is
join
last
less
lessEquals
limit
match
max
maxBy
min
minBy
mselect
none
not
notBetween
notBetweenEquals
notContains
notEmpty
notEnds
notEquals
notGreater
notGreaterEquals
notIs
notLess
notLessEquals
notMatch
notStarts
notType
notWhere
or
orBegin
orBetween
orBetweenEquals
orContains
orEmpty
orEnds
orEquals
orGreater
orGreaterEquals
orIs
orLess
orLessEquals
orMatch
orNot
orNotBetween
orNotBetweenEquals
orNotContains
orNotEmpty
orNotEnds
orNotEquals
orNotGreater
orNotGreaterEquals
orNotIs
orNotLess
orNotLessEquals
orNotMatch
orNotStarts
orNotType
orNotWhere
orStarts
orType
orWhere
pselect
query
removed
reverse
select
setWhere
skip
skipTake
skipWhile
sort
starts
stream
streamFn
sum
sumBy
tail
take
takeWhile
toDate
type
union
useCase
where

🤔 SQL

You can use simple SQL or H2 SQL. Althought you don’t need to refer the table the data can be refered from the ‘_TMP’ table.

Examples

# Simple SQL getting specific fields and ordering by one of them

curl -s https://api.github.com/repos/openaf/openaf/releases | oafp sql="select name, tag_name, published_at order by published_at" output=ctable

Using Path, From and/or SQL at the same time

Using path and from

# Using "path" to rename fields and then using "from" to limit the number of records

curl -s https://api.github.com/repos/openaf/openaf/releases | oafp path="[].{version:name, description:body}" from="limit(3)"

Use path to select a markdown field and parse it

# Get just the markdown body of the latest release and parsing it

curl -s https://api.github.com/repos/openaf/openaf/releases | oafp path="[0].body" output=md

Use path projections and SQL aggregation

# Use path to rename fields and the SQL to group by category of drink

curl -s "https://www.thecocktaildb.com/api/json/v1/1/search.php?s=martini" | oafp path="drinks[].{drink:strDrink,category:strCategory,alchool:strAlcoholic}" sql="select \"category\", count(1) \"count\" group by \"category\"" output=ctable