LuaJIT 学习（2）—— 使用 FFI 库的几个例子

介绍

The FFI library allows calling external C functions and using C data structures from pure Lua code.

The FFI library largely obviates the need to write tedious manual Lua/C bindings in C. No need to learn a separate binding language — it parses plain C declarations! These can be cut-n-pasted from C header files or reference manuals. It’s up to the task of binding large libraries without the need for dealing with fragile binding generators.

The FFI library is tightly integrated into LuaJIT (it’s not available as a separate module). The code generated by the JIT-compiler for accesses to C data structures from Lua code is on par with the code a C compiler would generate. Calls to C functions can be inlined in JIT-compiled code, unlike calls to functions bound via the classic Lua/C API.

Motivating Example: Calling External C Functions

It’s really easy to call an external C library function:

-- 1
local ffi = require("ffi")
-- 2
ffi.cdef[[
int printf(const char *fmt, ...);
]]
-- 3
ffi.C.printf("Hello %s!", "world")

1. Load the FFI library.
2. Add a C declaration for the function. The part inside the double-brackets is just standard C syntax.
3. Call the named C function — Yes, it’s that simple!

Actually, what goes on behind the scenes is far from simple: makes use of the standard C library namespace ffi.C. Indexing this namespace with a symbol name ("printf") automatically binds it to the standard C library. The result is a special kind of object which, when called, runs the printf function. The arguments passed to this function are automatically converted from Lua objects to the corresponding C types.

Ok, so maybe the use of printf() wasn’t such a spectacular example. You could have done that with io.write() and string.format(), too. But you get the idea …

So here’s something to pop up a message box on Windows:

local ffi = require("ffi")
ffi.cdef[[
int MessageBoxA(void *w, const char *txt, const char *cap, int type);
]]
ffi.C.MessageBoxA(nil, "Hello world!", "Test", 0)

Bing! Again, that was far too easy, no?

例子：Lua 中调用 C 函数

如果没有 FFI 库，调用 C 库的 printf 函数需要写一个 C 模块，然后在 Lua 代码中调用。

#include <stdio.h>
#include <stdarg.h>
#include <lua.h>
#include <lauxlib.h>

// C 函数：调用 printf 并返回打印的字符数
static int l_myprintf(lua_State *L) {
    const char *format = luaL_checkstring(L, 1); // 获取格式化字符串
    const char *str = luaL_checkstring(L, 2);    // 获取第二个参数

    int result = printf(format, str); // 调用 printf
    return 0; // 不返回任何 Lua 值
}

// Lua 调用 C 函数的映射表
static const struct luaL_Reg mylib[] = {
    {"printf", l_myprintf},
    {NULL, NULL} // 结束标记
};

// 入口函数，注册库
int luaopen_mylib(lua_State *L) {
    luaL_register(L, "mylib", mylib);
    return 1;
}

Lua 代码

local mylib = require("mylib")
mylib.printf("Hello %s!\n", "world")

Compare this with the effort required to bind that function using the classic Lua/C API: create an extra C file, add a C function that retrieves and checks the argument types passed from Lua and calls the actual C function, add a list of module functions and their names, add a luaopen_* function and register all module functions, compile and link it into a shared library (DLL), move it to the proper path, add Lua code that loads the module aaaand … finally call the binding function. Phew!

一对比，调用 C 函数简单了很多！

再看一个例子，在 Lua 中直接使用 C 语言的结构体。

Motivating Example: Using C Data Structures

The FFI library allows you to create and access C data structures. Of course, the main use for this is for interfacing with C functions. But they can be used stand-alone, too.

Here’s a sketch of a library that operates on color images, plus a simple benchmark. First, the plain Lua version:

local floor = math.floor

local function image_ramp_green(n)
  local img = {}
  local f = 255/(n-1)
  for i=1,n do
    img[i] = { red = 0, green = floor((i-1)*f), blue = 0, alpha = 255 }
  end
  return img
end

local function image_to_gray(img, n)
  for i=1,n do
    local y = floor(0.3*img[i].red + 0.59*img[i].green + 0.11*img[i].blue)
    img[i].red = y; img[i].green = y; img[i].blue = y
  end
end

local N = 400*400
local img = image_ramp_green(N)
for i=1,1000 do
  image_to_gray(img, N)
end

This creates a table with 160.000 pixels, each of which is a table holding four number values in the range of 0-255. First, an image with a green ramp is created (1D for simplicity), then the image is converted to grayscale 1000 times. Yes, that’s silly, but I was in need of a simple example …

And here’s the FFI version. The modified parts have been marked in bold:

-- 1
local ffi = require("ffi")
ffi.cdef[[
typedef struct { uint8_t red, green, blue, alpha; } rgba_pixel;
]]

local function image_ramp_green(n)
-- 2
  local img = ffi.new("rgba_pixel[?]", n)
  local f = 255/(n-1)
-- 3
  for i=0,n-1 do
-- 4
    img[i].green = i*f
    img[i].alpha = 255
  end
  return img
end

local function image_to_grey(img, n)
-- 3
  for i=0,n-1 do
-- 5
    local y = 0.3*img[i].red + 0.59*img[i].green + 0.11*img[i].blue
    img[i].red = y; img[i].green = y; img[i].blue = y
  end
end

local N = 400*400
local img = image_ramp_green(N)
for i=1,1000 do
  image_to_grey(img, N)
end

Ok, so that wasn’t too difficult:

1. First, load the FFI library and declare the low-level data type. Here we choose a struct which holds four byte fields, one for each component of a 4x8 bit RGBA pixel.
2. Creating the data structure with ffi.new() is straightforward — the '?' is a placeholder for the number of elements of a variable-length array.
3. C arrays are zero-based, so the indexes have to run from 0 to n-1. One might want to allocate one more element instead to simplify converting legacy code.
4. Since ffi.new() zero-fills the array by default, we only need to set the green and the alpha fields.
5. The calls to math.floor() can be omitted here, because floating-point numbers are already truncated towards zero when converting them to an integer. This happens implicitly when the number is stored in the fields of each pixel.

Now let’s have a look at the impact of the changes: first, memory consumption for the image is down from 22 Megabytes to 640 Kilobytes (4004004 bytes). That’s a factor of 35x less! So, yes, tables do have a noticeable overhead. BTW: The original program would consume 40 Megabytes in plain Lua (on x64).

内存消耗比原始的程序少35倍！

Next, performance: the pure Lua version runs in 9.57 seconds (52.9 seconds with the Lua interpreter) and the FFI version runs in 0.48 seconds on my machine (YMMV). That’s a factor of 20x faster (110x faster than the Lua interpreter).

使用 LuaJIT FFI 库，执行时间比使用 Lua5.1 的纯 Lua 程序快110倍！

知道 LuaJIT 牛，没想到这么牛！

再看一些例子

Accessing Standard System Functions

The following code explains how to access standard system functions. We slowly print two lines of dots by sleeping for 10 milliseconds after each dot:

local ffi = require("ffi")
-- 1
ffi.cdef[[
void Sleep(int ms);
int poll(struct pollfd *fds, unsigned long nfds, int timeout);
]]

local sleep
-- 2
if ffi.os == "Windows" then
-- 3
  function sleep(s)
-- 4
    ffi.C.Sleep(s*1000)
  end
else
  function sleep(s)
-- 5
    ffi.C.poll(nil, 0, s*1000)
  end
end

for i=1,160 do
  io.write("."); io.flush()
-- 6
  sleep(0.01)
end
io.write("\n")

Here’s the step-by-step explanation:

1. This defines the C library functions we’re going to use. The part inside the double-brackets is just standard C syntax. You can usually get this info from the C header files or the documentation provided by each C library or C compiler.
2. The difficulty we’re facing here, is that there are different standards to choose from. Windows has a simple Sleep() function. On other systems there are a variety of functions available to achieve sub-second sleeps, but with no clear consensus. Thankfully poll() can be used for this task, too, and it’s present on most non-Windows systems. The check for ffi.os makes sure we use the Windows-specific function only on Windows systems.
3. Here we’re wrapping the call to the C function in a Lua function. This isn’t strictly necessary, but it’s helpful to deal with system-specific issues only in one part of the code. The way we’re wrapping it ensures the check for the OS is only done during initialization and not for every call.
4. A more subtle point is that we defined our sleep() function (for the sake of this example) as taking the number of seconds, but accepting fractional seconds. Multiplying this by 1000 gets us milliseconds, but that still leaves it a Lua number, which is a floating-point value. Alas, the Sleep() function only accepts an integer value. Luckily for us, the FFI library automatically performs the conversion when calling the function (truncating the FP value towards zero, like in C).

Some readers will notice that Sleep() is part of KERNEL32.DLL and is also a stdcall function. So how can this possibly work? The FFI library provides the ffi.C default C library namespace, which allows calling functions from the default set of libraries, like a C compiler would. Also, the FFI library automatically detects stdcall functions, so you don’t need to declare them as such.

5. The poll() function takes a couple more arguments we’re not going to use. You can simply use nil to pass a NULL pointer and 0 for the nfds parameter. Please note, that the number 0 does not convert to a pointer value, unlike in C++. You really have to pass pointers to pointer arguments and numbers to number arguments.

The page on FFI semantics has all of the gory details about conversions between Lua objects and C types. For the most part you don’t have to deal with this, as it’s performed automatically and it’s carefully designed to bridge the semantic differences between Lua and C.

6. Now that we have defined our own sleep() function, we can just call it from plain Lua code. That wasn’t so bad, huh? Turning these boring animated dots into a fascinating best-selling game is left as an exercise for the reader. 😃

Accessing the zlib Compression Library

The following code shows how to access the zlib compression library from Lua code. We’ll define two convenience wrapper functions that take a string and compress or uncompress it to another string:

local ffi = require("ffi")
-- 1
ffi.cdef[[
unsigned long compressBound(unsigned long sourceLen);
int compress2(uint8_t *dest, unsigned long *destLen,
	      const uint8_t *source, unsigned long sourceLen, int level);
int uncompress(uint8_t *dest, unsigned long *destLen,
	       const uint8_t *source, unsigned long sourceLen);
]]
-- 2
local zlib = ffi.load(ffi.os == "Windows" and "zlib1" or "z")

local function compress(txt)
-- 3
  local n = zlib.compressBound(#txt)
  local buf = ffi.new("uint8_t[?]", n)
-- 4
  local buflen = ffi.new("unsigned long[1]", n)
  local res = zlib.compress2(buf, buflen, txt, #txt, 9)
  assert(res == 0)
-- 5
  return ffi.string(buf, buflen[0])
end

-- 6
local function uncompress(comp, n)
  local buf = ffi.new("uint8_t[?]", n)
  local buflen = ffi.new("unsigned long[1]", n)
  local res = zlib.uncompress(buf, buflen, comp, #comp)
  assert(res == 0)
  return ffi.string(buf, buflen[0])
end

-- 7. Simple test code.
local txt = string.rep("abcd", 1000)
print("Uncompressed size: ", #txt)
local c = compress(txt)
print("Compressed size: ", #c)
local txt2 = uncompress(c, #txt)
assert(txt2 == txt)

Here’s the step-by-step explanation:

1. This defines some of the C functions provided by zlib. For the sake of this example, some type indirections have been reduced and it uses the predefined fixed-size integer types, while still adhering to the zlib API/ABI.

2. This loads the zlib shared library. On POSIX systems, it’s named libz.so and usually comes pre-installed. Since ffi.load() automatically adds any missing standard prefixes/suffixes, we can simply load the "z" library. On Windows it’s named zlib1.dll and you’ll have to download it first from the zlib site. The check for ffi.os makes sure we pass the right name to ffi.load().

3. First, the maximum size of the compression buffer is obtained by calling the zlib.compressBound function with the length of the uncompressed string. The next line allocates a byte buffer of this size. The [?] in the type specification indicates a variable-length array (VLA). The actual number of elements of this array is given as the 2nd argument to ffi.new().

4. This may look strange at first, but have a look at the declaration of the compress2 function from zlib: the destination length is defined as a pointer! This is because you pass in the maximum buffer size and get back the actual length that was used.

   In C you’d pass in the address of a local variable (&buflen). But since there’s no address-of operator in Lua, we’ll just pass in a one-element array. Conveniently, it can be initialized with the maximum buffer size in one step. Calling the actual zlib.compress2 function is then straightforward.

5. We want to return the compressed data as a Lua string, so we’ll use ffi.string(). It needs a pointer to the start of the data and the actual length. The length has been returned in the buflen array, so we’ll just get it from there.

Note that since the function returns now, the buf and buflen variables will eventually be garbage collected. This is fine, because ffi.string() has copied the contents to a newly created (interned) Lua string. If you plan to call this function lots of times, consider reusing the buffers and/or handing back the results in buffers instead of strings. This will reduce the overhead for garbage collection and string interning.

6. The uncompress functions does the exact opposite of the compress function. The compressed data doesn’t include the size of the original string, so this needs to be passed in. Otherwise, no surprises here.
7. The code, that makes use of the functions we just defined, is just plain Lua code. It doesn’t need to know anything about the LuaJIT FFI — the convenience wrapper functions completely hide it.

One major advantage of the LuaJIT FFI is that you are now able to write those wrappers in Lua. And at a fraction of the time it would cost you to create an extra C module using the Lua/C API. Many of the simpler C functions can probably be used directly from your Lua code, without any wrappers.

Side note: the zlib API uses the long type for passing lengths and sizes around. But all those zlib functions actually only deal with 32 bit values. This is an unfortunate choice for a public API, but may be explained by zlib’s history — we’ll just have to deal with it.

First, you should know that a long is a 64 bit type e.g. on POSIX/x64 systems, but a 32 bit type on Windows/x64 and on 32 bit systems. Thus a long result can be either a plain Lua number or a boxed 64 bit integer cdata object, depending on the target system.

Ok, so the ffi.* functions generally accept cdata objects wherever you’d want to use a number. That’s why we get a away with passing n to ffi.string() above. But other Lua library functions or modules don’t know how to deal with this. So for maximum portability, one needs to use tonumber() on returned long results before passing them on. Otherwise the application might work on some systems, but would fail in a POSIX/x64 environment.

Defining Metamethods for a C Type

The following code explains how to define metamethods for a C type. We define a simple point type and add some operations to it:

local ffi = require("ffi")
-- 1
ffi.cdef[[
typedef struct { double x, y; } point_t;
]]

-- 2
local point
local mt = {
-- 3
  __add = function(a, b) return point(a.x+b.x, a.y+b.y) end,
  __len = function(a) return math.sqrt(a.x*a.x + a.y*a.y) end,
-- 4
  __index = {
    area = function(a) return a.x*a.x + a.y*a.y end,
  },
}
-- 5
point = ffi.metatype("point_t", mt)

-- 6
local a = point(3, 4)
print(a.x, a.y)  --> 3  4
print(#a)        --> 5
print(a:area())  --> 25
local b = a + point(0.5, 8)
print(#b)        --> 12.5

Here’s the step-by-step explanation:

1. This defines the C type for a two-dimensional point object.
2. We have to declare the variable holding the point constructor first, because it’s used inside of a metamethod.
3. Let’s define an __add metamethod which adds the coordinates of two points and creates a new point object. For simplicity, this function assumes that both arguments are points. But it could be any mix of objects, if at least one operand is of the required type (e.g. adding a point plus a number or vice versa). Our __len metamethod returns the distance of a point to the origin.
4. If we run out of operators, we can define named methods, too. Here, the __index table defines an area function. For custom indexing needs, one might want to define __index and __newindex functions instead.
5. This associates the metamethods with our C type. This only needs to be done once. For convenience, a constructor is returned by ffi.metatype(). We’re not required to use it, though. The original C type can still be used e.g. to create an array of points. The metamethods automatically apply to any and all uses of this type.

Please note, that the association with a metatable is permanent and the metatable must not be modified afterwards! Ditto for the __index table.

6. Here are some simple usage examples for the point type and their expected results. The predefined operations (such as a.x) can be freely mixed with the newly defined metamethods. Note that area is a method and must be called with the Lua syntax for methods: a:area(), not a.area().

The C type metamethod mechanism is most useful when used in conjunction with C libraries that are written in an object-oriented style. Creators return a pointer to a new instance, and methods take an instance pointer as the first argument. Sometimes you can just point __index to the library namespace and __gc to the destructor and you’re done. But often enough you’ll want to add convenience wrappers, e.g. to return actual Lua strings or when returning multiple values.

举个例子，如果你有一个 C 库 mylib，它有一个方法 mylib:do_something()，你可以通过 __index 将该方法暴露给 Lua：

local ffi = require("ffi")
ffi.cdef[[
  typedef struct { int x, y; } MyObject;
  void do_something(MyObject* obj);
]]

local mylib = ffi.load("mylib")

-- 绑定一个 MyObject 类型
local MyObject = ffi.metatype("MyObject", {
  __index = mylib,  -- 将 __index 指向库的命名空间
})

-- 创建一个 MyObject 实例
local obj = MyObject()

-- 直接使用 C 库的方法
obj:do_something()

文档中还提到，有时候你需要手动定义 __gc 来处理对象的销毁。在 Lua 中创建的 C 对象，可能需要在 Lua 对象被垃圾回收时清理对应的 C 资源（例如关闭文件句柄、释放内存等）。你可以通过 __gc 来定义析构函数。

local ffi = require("ffi")

ffi.cdef[[
  typedef struct { int x, y; } MyObject;
  void destroy_object(MyObject* obj);
]]

local mylib = ffi.load("mylib")

local MyObject = ffi.metatype("MyObject", {
  -- 定义析构函数
  __gc = function(self)
    mylib.destroy_object(self)
  end
})

-- 创建一个 MyObject 实例
local obj = MyObject()

Some C libraries only declare instance pointers as an opaque void * type. In this case you can use a fake type for all declarations, e.g. a pointer to a named (incomplete) struct will do: typedef struct foo_type *foo_handle. The C side doesn’t know what you declare with the LuaJIT FFI, but as long as the underlying types are compatible, everything still works.

例子：创建 userdata

如果不使用 LuaJIT，使用标准的 Lua 解释器实现相同的程序，那么需要创建一个 C 模块，模块提供函数创建一个 userdata，并给它设置元表

#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>
#include <math.h>
#include <stdlib.h>

typedef struct {
    double x, y;
} point_t;

// 创建 point 实例
static int point_new(lua_State *L) {
    double x = luaL_checknumber(L, 1);
    double y = luaL_checknumber(L, 2);

    point_t *p = (point_t *)lua_newuserdata(L, sizeof(point_t));
    p->x = x;
    p->y = y;

    luaL_getmetatable(L, "PointMT");
    lua_setmetatable(L, -2);
    return 1;
}

// __add 运算符
static int point_add(lua_State *L) {
    point_t *a = (point_t *)luaL_checkudata(L, 1, "PointMT");
    point_t *b = (point_t *)luaL_checkudata(L, 2, "PointMT");

    lua_pushcfunction(L, point_new);
    lua_pushnumber(L, a->x + b->x);
    lua_pushnumber(L, a->y + b->y);
    lua_call(L, 2, 1);

    return 1;
}

// __len 运算符
static int point_len(lua_State *L) {
    point_t *a = (point_t *)luaL_checkudata(L, 1, "PointMT");
    lua_pushnumber(L, sqrt(a->x * a->x + a->y * a->y));
    return 1;
}

// area 方法
static int point_area(lua_State *L) {
    point_t *a = (point_t *)luaL_checkudata(L, 1, "PointMT");
    lua_pushnumber(L, a->x * a->x + a->y * a->y);
    return 1;
}

// 获取字段
static int point_get(lua_State *L) {
    point_t *p = (point_t *)luaL_checkudata(L, 1, "PointMT");
    const char *key = luaL_checkstring(L, 2);

    if (strcmp(key, "x") == 0) {
        lua_pushnumber(L, p->x);
    } else if (strcmp(key, "y") == 0) {
        lua_pushnumber(L, p->y);
    } else if (strcmp(key, "area") == 0) {
        lua_pushcfunction(L, point_area);
    } else {
        lua_pushnil(L);
    }
    return 1;
}

// 注册 point_t 相关的元表和方法
static void create_point_metatable(lua_State *L) {
    luaL_newmetatable(L, "PointMT");

    lua_pushstring(L, "__add");
    lua_pushcfunction(L, point_add);
    lua_settable(L, -3);

    lua_pushstring(L, "__len");
    lua_pushcfunction(L, point_len);
    lua_settable(L, -3);

    lua_pushstring(L, "__index");
    lua_pushcfunction(L, point_get);
    lua_settable(L, -3);

    lua_pop(L, 1);
}

// 注册模块
static const luaL_Reg pointlib[] = {
    {"new", point_new},
    {NULL, NULL}
};

int luaopen_point(lua_State *L) {
    create_point_metatable(L);
    luaL_register(L, "point", pointlib);
    return 1;
}

然后在 Lua 中使用这个模块

local point = require("point")

local a = point.new(3, 4)
print(a.x, a.y)   -- 输出 3  4
print(#a)         -- 输出 5
print(a:area())   -- 输出 25

local b = a + point.new(0.5, 8)
print(#b)         -- 输出 12.5

整个流程要复杂很多。不得不说 LuaJIT 真牛逼！

Translating C Idioms

Here’s a list of common C idioms and their translation to the LuaJIT FFI:

To Cache or Not to Cache

The JIT compiler has special logic to eliminate all of the lookup overhead for functions resolved from a C library namespace! Thus it’s not helpful and actually counter-productive to cache individual C functions like this:

local funca, funcb = ffi.C.funca, ffi.C.funcb -- Not helpful!
local function foo(x, n)
  for i=1,n do funcb(funca(x, i), 1) end
end

This turns them into indirect calls and generates bigger and slower machine code. Instead, you’ll want to cache the namespace itself and rely on the JIT compiler to eliminate the lookups:

local C = ffi.C          -- Instead use this!
local function foo(x, n)
  for i=1,n do C.funcb(C.funca(x, i), 1) end
end

This generates both shorter and faster code. So don’t cache C functions, but do cache namespaces! Most often the namespace is already in a local variable at an outer scope, e.g. from local lib = ffi.load(…). Note that copying it to a local variable in the function scope is unnecessary.