Skip to main content

Specification

The purpose of this document is to spec TypeStream language features.

While TypeStream syntax is inspired by bash, feature parity is a non goal of the project.

Keywords​

TODO

Types​

By default, TypeStream doesn't need a syntax to specify types. It can infer types from context.

  • String
  • Number
  • List
  • DataStream

DataStream​

In TypeStream, a DataStream is a typed source of data. TypeStream always outputs structured data (defaulting to encoding data in JSON).

Let's start from some examples:

# words is a "string" stream, meaning each message value is a string

$ cat words # 1,"first word" 2,"second word"

# users is a "record" topic, meaning each message value is structured record (can be avro, json, protobuf, etc..)

$ cat users # 1, [name: Foo, age: 42] 2, [name: Bar, age: 24]

# what happens if we join users and words?

$ join users words

# 1, [users: {name: Foo, age: 42}, words: "first word"]
# 2, [users: {name: Foo, age: 42}, words: "second word"]

Merging data streams​

Block expression​

Block expressions define anonymous functions that take one parameter. They're used by TypeStream data operators like each and enrich.

Here's their syntax:

{ <var> -> <pipeline> }

and an example using the each operator:

cat users | each { user -> echo $user }

In the example above, the user variable is bound to each record in the users data stream.