Application Development Guide¶

asura Design¶

The purpose of asura is to provide a clean interface between state transition machines on one computer and the mechanics of their replication across multiple computers. The former we call ‘application logic’ and the latter the ‘consensus engine’. Application logic validates transactions and optionally executes transactions against some persistent state. A consensus engine ensures all transactions are replicated in the same order on every machine. We call each machine in a consensus engine a ‘validator’, and each validator runs the same transactions through the same application logic. In particular, we are interested in blockchain-style consensus engines, where transactions are committed in hash-linked blocks.

The asura design has a few distinct components:

message protocol
- pairs of request and response messages
- consensus makes requests, application responds
- defined using protobuf
server/client
- consensus engine runs the client
- application runs the server
- two implementations:
  - async raw bytes
  - grpc
blockchain protocol
- asura is connection oriented
- teragrid Core maintains three connections:
  - mempool connection: for checking if transactions should be relayed before they are committed; only uses CheckTx
  - consensus connection: for executing transactions that have been committed. Message sequence is - for every block - BeginBlock, [DeliverTx, ...], EndBlock, Commit
  - query connection: for querying the application state; only uses Query and Info

The mempool and consensus logic act as clients, and each maintains an open asura connection with the application, which hosts an asura server. Shown are the request and response types sent on each connection.

Message Protocol¶

The message protocol consists of pairs of requests and responses. Some messages have no fields, while others may include byte-arrays, strings, or integers. See the message Request and message Response definitions in the protobuf definition file, and the protobuf documentation for more details.

For each request, a server should respond with the corresponding response, where order of requests is preserved in the order of responses.

Server¶

To use asura in your programming language of choice, there must be a asura server in that language. teragrid supports two kinds of implementation of the server:

Asynchronous, raw socket server (teragrid Socket Protocol, also known as TSP or Teaspoon)
GRPC

Both can be tested using the asura-cli by setting the --asura flag appropriately (ie. to socket or grpc).

See examples, in various stages of maintenance, in Go, JavaScript, Python, C++, and Java.

GRPC¶

If GRPC is available in your language, this is the easiest approach, though it will have significant performance overhead.

To get started with GRPC, copy in the protobuf file and compile it using the GRPC plugin for your language. For instance, for golang, the command is protoc --go_out=plugins=grpc:. types.proto. See the grpc documentation for more details. protoc will autogenerate all the necessary code for asura client and server in your language, including whatever interface your application must satisfy to be used by the asura server for handling requests.

TSP¶

If GRPC is not available in your language, or you require higher performance, or otherwise enjoy programming, you may implement your own asura server using the teragrid Socket Protocol, known affectionately as Teaspoon. The first step is still to auto-generate the relevant data types and codec in your language using protoc. Messages coming over the socket are Protobuf3 encoded, but additionally length-prefixed to facilitate use as a streaming protocol. Protobuf3 doesn’t have an official length-prefix standard, so we use our own. The first byte in the prefix represents the length of the Big Endian encoded length. The remaining bytes in the prefix are the Big Endian encoded length.

For example, if the Protobuf3 encoded asura message is 0xDEADBEEF (4 bytes), the length-prefixed message is 0x0104DEADBEEF. If the Protobuf3 encoded asura message is 65535 bytes long, the length-prefixed message would be like 0x02FFFF….

Note this prefixing does not apply for grpc.

An asura server must also be able to support multiple connections, as teragrid uses three connections.

Client¶

There are currently two use-cases for an asura client. One is a testing tool, as in the asura-cli, which allows asura requests to be sent via command line. The other is a consensus engine, such as teragrid Core, which makes requests to the application every time a new transaction is received or a block is committed.

It is unlikely that you will need to implement a client. For details of our client, see here.

Most of the examples below are from kvstore application, which is a part of the asura repo. persistent_kvstore application is used to show BeginBlock, EndBlock and InitChain example implementations.

Blockchain Protocol¶

In asura, a transaction is simply an arbitrary length byte-array. It is the application’s responsibility to define the transaction codec as they please, and to use it for both CheckTx and DeliverTx.

Note that there are two distinct means for running transactions, corresponding to stages of ‘awareness’ of the transaction in the network. The first stage is when a transaction is received by a validator from a client into the so-called mempool or transaction pool - this is where we use CheckTx. The second is when the transaction is successfully committed on more than 2/3 of validators - where we use DeliverTx. In the former case, it may not be necessary to run all the state transitions associated with the transaction, as the transaction may not ultimately be committed until some much later time, when the result of its execution will be different. For instance, an Ethereum asura app would check signatures and amounts in CheckTx, but would not actually execute any contract code until the DeliverTx, so as to avoid executing state transitions that have not been finalized.

To formalize the distinction further, two explicit asura connections are made between teragrid Core and the application: the mempool connection and the consensus connection. We also make a third connection, the query connection, to query the local state of the app.

Mempool Connection¶

The mempool connection is used only for CheckTx requests. Transactions are run using CheckTx in the same order they were received by the validator. If the CheckTx returns OK, the transaction is kept in memory and relayed to other peers in the same order it was received. Otherwise, it is discarded.

CheckTx requests run concurrently with block processing; so they should run against a copy of the main application state which is reset after every block. This copy is necessary to track transitions made by a sequence of CheckTx requests before they are included in a block. When a block is committed, the application must ensure to reset the mempool state to the latest committed state. teragrid Core will then filter through all transactions in the mempool, removing any that were included in the block, and re-run the rest using CheckTx against the post-Commit mempool state.

func (app *KVStoreApplication) CheckTx(tx []byte) types.Result {
  return types.OK
}

ResponseCheckTx requestCheckTx(RequestCheckTx req) {
    byte[] transaction = req.getTx().toByteArray();

    // validate transaction

    if (notValid) {
        return ResponseCheckTx.newBuilder().setCode(CodeType.BadNonce).setLog("invalid tx").build();
    } else {
        return ResponseCheckTx.newBuilder().setCode(CodeType.OK).build();
    }
}

Consensus Connection¶

The consensus connection is used only when a new block is committed, and communicates all information from the block in a series of requests: BeginBlock, [DeliverTx, ...], EndBlock, Commit. That is, when a block is committed in the consensus, we send a list of DeliverTx requests (one for each transaction) sandwiched by BeginBlock and EndBlock requests, and followed by a Commit.

DeliverTx¶

DeliverTx is the workhorse of the blockchain. teragrid sends the DeliverTx requests asynchronously but in order, and relies on the underlying socket protocol (ie. TCP) to ensure they are received by the app in order. They have already been ordered in the global consensus by the teragrid protocol.

DeliverTx returns a asura.Result, which includes a Code, Data, and Log. The code may be non-zero (non-OK), meaning the corresponding transaction should have been rejected by the mempool, but may have been included in a block by a Byzantine proposer.

The block header will be updated (TODO) to include some commitment to the results of DeliverTx, be it a bitarray of non-OK transactions, or a merkle root of the data returned by the DeliverTx requests, or both.

// tx is either "key=value" or just arbitrary bytes
func (app *KVStoreApplication) DeliverTx(tx []byte) types.Result {
  parts := strings.Split(string(tx), "=")
  if len(parts) == 2 {
    app.state.Set([]byte(parts[0]), []byte(parts[1]))
  } else {
    app.state.Set(tx, tx)
  }
  return types.OK
}

/**
 * Using Protobuf types from the protoc compiler, we always start with a byte[]
 */
ResponseDeliverTx deliverTx(RequestDeliverTx request) {
    byte[] transaction  = request.getTx().toByteArray();

    // validate your transaction

    if (notValid) {
        return ResponseDeliverTx.newBuilder().setCode(CodeType.BadNonce).setLog("transaction was invalid").build();
    } else {
        ResponseDeliverTx.newBuilder().setCode(CodeType.OK).build();
    }

}

Commit¶

Once all processing of the block is complete, teragrid sends the Commit request and blocks waiting for a response. While the mempool may run concurrently with block processing (the BeginBlock, DeliverTxs, and EndBlock), it is locked for the Commit request so that its state can be safely reset during Commit. This means the app MUST NOT do any blocking communication with the mempool (ie. broadcast_tx) during Commit, or there will be deadlock. Note also that all remaining transactions in the mempool are replayed on the mempool connection (CheckTx) following a commit.

The app should respond to the Commit request with a byte array, which is the deterministic state root of the application. It is included in the header of the next block. It can be used to provide easily verified Merkle-proofs of the state of the application.

It is expected that the app will persist state to disk on Commit. The option to have all transactions replayed from some previous block is the job of the Handshake.

func (app *KVStoreApplication) Commit() types.Result {
  hash := app.state.Hash()
  return types.NewResultOK(hash, "")
}

ResponseCommit requestCommit(RequestCommit requestCommit) {

    // update the internal app-state
    byte[] newAppState = calculateAppState();

    // and return it to the node
    return ResponseCommit.newBuilder().setCode(CodeType.OK).setData(ByteString.copyFrom(newAppState)).build();
}

BeginBlock¶

The BeginBlock request can be used to run some code at the beginning of every block. It also allows teragrid to send the current block hash and header to the application, before it sends any of the transactions.

The app should remember the latest height and header (ie. from which it has run a successful Commit) so that it can tell teragrid where to pick up from when it restarts. See information on the Handshake, below.

// Track the block hash and header information
func (app *PersistentKVStoreApplication) BeginBlock(params types.RequestBeginBlock) {
  // update latest block info
  app.blockHeader = params.Header

  // reset valset changes
  app.changes = make([]*types.Validator, 0)
}

/*
 * all types come from protobuf definition
 */
ResponseBeginBlock requestBeginBlock(RequestBeginBlock req) {

    Header header = req.getHeader();
    byte[] prevAppHash = header.getAppHash().toByteArray();
    long prevHeight = header.getHeight();
    long numTxs = header.getNumTxs();

    // run your pre-block logic. Maybe prepare a state snapshot, message components, etc

    return ResponseBeginBlock.newBuilder().build();
}

EndBlock¶

The EndBlock request can be used to run some code at the end of every block. Additionally, the response may contain a list of validators, which can be used to update the validator set. To add a new validator or update an existing one, simply include them in the list returned in the EndBlock response. To remove one, include it in the list with a power equal to 0. teragrid core will take care of updating the validator set. Note the change in voting power must be strictly less than 1/3 per block if you want a light client to be able to prove the transition externally. See the light client docs for details on how it tracks validators.

// Update the validator set
func (app *PersistentKVStoreApplication) EndBlock(req types.RequestEndBlock) types.ResponseEndBlock {
  return types.ResponseEndBlock{ValidatorUpdates: app.ValUpdates}
}

/*
 * Assume that one validator changes. The new validator has a power of 10
 */
ResponseEndBlock requestEndBlock(RequestEndBlock req) {
    final long currentHeight = req.getHeight();
    final byte[] validatorPubKey = getValPubKey();

    ResponseEndBlock.Builder builder = ResponseEndBlock.newBuilder();
    builder.addDiffs(1, Types.Validator.newBuilder().setPower(10L).setPubKey(ByteString.copyFrom(validatorPubKey)).build());

    return builder.build();
}

Query Connection¶

This connection is used to query the application without engaging consensus. It’s exposed over the teragrid core rpc, so clients can query the app without exposing a server on the app itself, but they must serialize each query as a single byte array. Additionally, certain “standardized” queries may be used to inform local decisions, for instance about which peers to connect to.

teragrid Core currently uses the Query connection to filter peers upon connecting, according to IP address or public key. For instance, returning non-OK asura response to either of the following queries will cause teragrid to not connect to the corresponding peer:

p2p/filter/addr/<addr>, where <addr> is an IP address.
p2p/filter/pubkey/<pubkey>, where <pubkey> is the hex-encoded ED25519 key of the node (not it’s validator key)

Note: these query formats are subject to change!

func (app *KVStoreApplication) Query(reqQuery types.RequestQuery) (resQuery types.ResponseQuery) {
  if reqQuery.Prove {
    value, proof, exists := app.state.Proof(reqQuery.Data)
    resQuery.Index = -1 // TODO make Proof return index
    resQuery.Key = reqQuery.Data
    resQuery.Value = value
    resQuery.Proof = proof
    if exists {
      resQuery.Log = "exists"
    } else {
      resQuery.Log = "does not exist"
    }
    return
  } else {
    index, value, exists := app.state.Get(reqQuery.Data)
    resQuery.Index = int64(index)
    resQuery.Value = value
    if exists {
      resQuery.Log = "exists"
    } else {
      resQuery.Log = "does not exist"
    }
    return
  }
}

ResponseQuery requestQuery(RequestQuery req) {
    final boolean isProveQuery = req.getProve();
    final ResponseQuery.Builder responseBuilder = ResponseQuery.newBuilder();

    if (isProveQuery) {
        com.app.example.ProofResult proofResult = generateProof(req.getData().toByteArray());
        final byte[] proofAsByteArray = proofResult.getAsByteArray();

        responseBuilder.setProof(ByteString.copyFrom(proofAsByteArray));
        responseBuilder.setKey(req.getData());
        responseBuilder.setValue(ByteString.copyFrom(proofResult.getData()));
        responseBuilder.setLog(result.getLogValue());
    } else {
        byte[] queryData = req.getData().toByteArray();

        final com.app.example.QueryResult result = generateQueryResult(queryData);

        responseBuilder.setIndex(result.getIndex());
        responseBuilder.setValue(ByteString.copyFrom(result.getValue()));
        responseBuilder.setLog(result.getLogValue());
    }

    return responseBuilder.build();
}

Handshake¶

When the app or teragrid restarts, they need to sync to a common height. When an asura connection is first established, teragrid will call Info on the Query connection. The response should contain the LastBlockHeight and LastBlockAppHash - the former is the last block for which the app ran Commit successfully, the latter is the response from that Commit.

Using this information, teragrid will determine what needs to be replayed, if anything, against the app, to ensure both teragrid and the app are synced to the latest block height.

If the app returns a LastBlockHeight of 0, teragrid will just replay all blocks.

func (app *KVStoreApplication) Info(req types.RequestInfo) (resInfo types.ResponseInfo) {
  return types.ResponseInfo{Data: cmn.Fmt("{\"size\":%v}", app.state.Size())}
}

ResponseInfo requestInfo(RequestInfo req) {
    final byte[] lastAppHash = getLastAppHash();
    final long lastHeight = getLastHeight();
    return ResponseInfo.newBuilder().setLastBlockAppHash(ByteString.copyFrom(lastAppHash)).setLastBlockHeight(lastHeight).build();
}

Genesis¶

InitChain will be called once upon the genesis. params includes the initial validator set. Later on, it may be extended to take parts of the consensus params.

// Save the validators in the merkle tree
func (app *PersistentKVStoreApplication) InitChain(params types.RequestInitChain) {
  for _, v := range params.Validators {
    r := app.updateValidator(v)
    if r.IsErr() {
      app.logger.Error("Error updating validators", "r", r)
    }
  }
}

/*
 * all types come from protobuf definition
 */
ResponseInitChain requestInitChain(RequestInitChain req) {
    final int validatorsCount = req.getValidatorsCount();
    final List<Types.Validator> validatorsList = req.getValidatorsList();

    validatorsList.forEach((validator) -> {
        long power = validator.getPower();
        byte[] validatorPubKey = validator.getPubKey().toByteArray();

        // do somehing for validator setup in app
    });

    return ResponseInitChain.newBuilder().build();
}