[RFC][GOLANG] TVM Golang Runtime Interface Discussion

[srkreddy1238]

Here briefing the golang interface design aspects.

Few golang concepts

• Usage of packages C and unsafe golang packages to go across golang boundary while accessing functions, types and pointer ..etc.
• Slices: Usage if golang slice where ever is possible to simplify application programming. Example being use slice of string instead of argument pair containing argc, argv

Please refer to golang tutorials for details of cgo, unsafe, slices ..etc.

Objective

• Initial version is expected to cover basic inference API listed below.
• Similar to many programming languages a package which can be imported and used is the best option here too.
• Hence the objective of this effort is a golang package gotvm which is wrapped over TVM c_runtime_api.

API

This initial version expected to cover the below API

func FuncListGlobalNames() (retVal []string, err error)
func GetGlobalFunction(funcname string) (retVal func(args ...interface{}) (*Value, error), err error)
func RegisterFunction(args ...interface{}) (err error)
type Array
    func Empty(shape []int64, args ...interface{}) (parray *Array, err error)
    func (parray Array) AsSlice() (retVal interface{}, err error)
    func (parray Array) CopyFrom(val interface{}) (err error)
    func (parray Array) GetCtx() (retVal Context)
    func (parray Array) GetDType() (retVal string)
    func (parray Array) GetNdim() (retVal int32)
    func (parray Array) GetShape() (retVal []int64)
type ByteArray
    func NewByteArray(val []uint8) (retVal *ByteArray)
    func (tbytearray ByteArray) GetData() (retVal string)
type Context
    func CPU(index int32) Context
    func CPUPinned(index int32) Context
    func GPU(index int32) Context
    func Metal(index int32) Context
    func OpenCL(index int32) Context
    func OpenGL(index int32) Context
    func ROCM(index int32) Context
    func SDAccel(index int32) Context
    func VPI(index int32) Context
    func Vulkan(index int32) Context
type Function
    func ConvertFunction(args ...interface{}) (fhandle Function, err error)
    func (tvmfunction Function) Invoke(args ...interface{}) (retVal *Value, err error)
type Module
    func LoadModuleFromFile(modpath string, args ...interface{}) (retVal *Module, err error)
    func (tvmmodule *Module) GetFunction(funcname string, args ...interface{}) (retVal func(args ...interface{}) (*Value, error), err error)
type Value
    func (tvmval Value) AsFloat64() (retVal float64)
    func (tvmval Value) AsFunction() (retVal *Function)
    func (tvmval Value) AsInt64() (retVal int64)
    func (tvmval Value) AsModule() (retVal *Module)
    func (tvmval Value) AsStr() (retVal string)

Sample Code

/*!
 *  Copyright (c) 2018 by Contributors
 * \brief Sample golang application deployment over tvm.
 * \file simple.go
 */

package main

import (
    "fmt"
    "runtime"
    "./gotvm"
)

// NNVM compiled model paths.
const (
    modLib    = "./deploy.so"
)

// main
func main() {
    // Welcome
    defer runtime.GC()
    fmt.Printf("TVM Go Interface : v%v\n", gotvm.GoTVMVersion)
    fmt.Printf("TVM Version   : v%v\n", gotvm.TVMVersion)
    fmt.Printf("DLPACK Version: v%v\n\n", gotvm.DLPackVersion)

    // Import tvm module (so)
    modp, _ := gotvm.LoadModuleFromFile(modLib)
    fmt.Printf("Module Imported\n")


    // Allocate Array for inputs and outputs.

    // Allocation by explicit type and context.
    tshapeIn  := []int64{4}
    inX, _ := gotvm.Empty(tshapeIn, "float32", gotvm.CPU(0))

    // Default allocation on CPU
    inY, _ := gotvm.Empty(tshapeIn, "float32")

    // Default allocation to type "float32" and on CPU
    out, _ := gotvm.Empty(tshapeIn)

    fmt.Printf("Input and Output Arrays allocated\n")

    // Fill Input Data : inX , inY
    inXSlice := []float32 {1, 2, 3, 4}
    inYSlice := []float32 {5, 6, 7, 8}

    // Copy the data on target memory through runtime CopyFrom api.
    inX.CopyFrom(inXSlice)
    inY.CopyFrom(inYSlice)

    fmt.Printf("X: %v\n", inXSlice)
    fmt.Printf("Y: %v\n", inYSlice)

    // Get function "myadd"
    funp, _ := modp.GetFunction("myadd")

    // Call function
    funp(inX, inY, out)

    fmt.Printf("Module function myadd executed\n")

    // Get the output tensor as an interface holding a slice through runtime CopyTo api.
    outSlice, _ := out.AsSlice()

    // Print results
    fmt.Printf("Result:%v\n", outSlice.([]float32))
}

@dmlc/tvm-team welcome to comment on this info.

Please refer to #1470 to have a look at initial effort.

[tqchen]

I hope we can have a healthy discussion on what high-level API that goes beyond C FFI style API can be provided. Specifically, if we look at tvm runtime API of java, js, python, we have specific object class for Module, Array, and Function which automatically manages their lifecycle and when necessary overloads the ops. I know little about go but from what I learnt, this should be possible

[tqchen]

Some things I think that are related. In python and js we use variadic interface to implement function invocation, it seems to be possible to implement a dynamically dispatched variadic function via
func function_name(a ...interface{})

• see https://blog.learngoprogramming.com/golang-variadic-funcs-how-to-patterns-369408f19085

[tqchen]

Point2: ideally we want to remove Free functions and automatically manage the objects, and always use Finalizer to trigger the free of the objects https://golang.org/pkg/runtime/#SetFinalizer

[tqchen]

Point3: from user's point of view, we may not need to have a TVMFunction object at all, the handle can be captured, but module.GetFunction can directly return a closure with signature:

func function_name(a ...interface{})

This way the function returned by module.GetFunction can be directly invoked as a function

[tqchen]

Please update the examples accordingly, once we have a clear interface proposal, we can proceed on implementation again

[srkreddy1238]

Updated the interface and example code above.
This version is based on

• variadic args for variable parameters
• Finalizers for automatic garbage collection
• Interfaces to send and receive arbitrary datatypes

The PR #1470 also updated accordingly for review.

[tqchen]

Three things:

• Try to (hide) the C APIs and remove them from API list
• Use return value style

mod = gotvm.LoadModuleFromFile("xx.dso");

• Please check if there is a common convention of code style guide and have a consistent code style for function naming etc.

[srkreddy1238]

@tqchen Sorry, I forgot to submit the edited sample code and final API interface above :)

Have a look now.

• Only above listed API and Types are exposed on golang side.
• Yes, used return value style as ret, err. Optional second return value is golang style of reporting error. Here I am returning TVMGetLastErr as golang error object.
• Golang goes with CamelCase and PascalCase based on exported and un exported types.

[tqchen]

• TVMType might not be necessary, most cases people are good with strings e.g. "float32"
• Need to pass in TVMContext in array creation instead of flag
• GetData is dangerous as it only works on CPU, instead, do copy API(backed by copy(from/to)bytes
  • Checkout js api and java api https://docs.tvm.ai/jsdoc/tvm.NDArray.html
  • AsArray
  • CopyFrom

[tqchen]

maybe @yzhliu can also chime in to comment on api design

[srkreddy1238]

Addressed above changes.

• Strings instead of TVMType
• TVMContext exposed to user
• Copy API's used instead of direct access to TVMArray data.

[tqchen]

@srkreddy1238 can you please also update the sample code on the first post?