lua-resty-couchbase

Lua couchbase client driver for the ngx_lua based on the cosocket API

Account

iqiyi

Repo

https://github.com/iqiyi/lua-resty-couchbase

$ opm get iqiyi/lua-resty-couchbase

Name

[!GitHub issues](https://github.com/iqiyi/lua-resty-couchbase/issues) [!GitHub forks](https://github.com/iqiyi/lua-resty-couchbase/network) [!GitHub stars](https://github.com/iqiyi/lua-resty-couchbase/stargazers) [!GitHub license](https://github.com/iqiyi/lua-resty-couchbase/blob/master/license.txt)

lua-resty-couchbase - Lua couchbase client driver for the ngx_lua based on the cosocket API.

It has been running stably on IQIYI Play Service for more than 5 years, and the online support QPS reaches 20W. Under the OpenResty architecture, CouchBase can be accessed directly from nginx.

使用cosocket纯lua实现的couchbase的client，已经在爱奇艺重要的播放服务稳定运行5年多，线上支持峰值QPS达到20W，在OpenResty架构下面，可以直接从nginx上面访问CouchBase.

Status

This library is considered production ready.

Description

This Lua library is a CouchBase client driver for the ngx_lua nginx module:

https://github.com/openresty/lua-nginx-module/#readme

This Lua library takes advantage of ngx_lua's cosocket API, which ensures 100% nonblocking behavior.

Note that at least ngx_lua 0.5.14 or OpenResty 1.2.1.14 is required.

Synopsis

        lua_package_path "/path/to/lua-resty-couchbase/lib/?.lua;;";
    
        lua_shared_dict ldict 10m;
    
        server {
            location /test {
                content_by_lua_block {
                    local cjson = require "cjson"
                    local couchbase = require "resty.couchbase"
    
                    local function get_from_service()
                        -- nothing
                        return "{}"
                    end
    
                    local conf = {
                        hosts = { "10.10.10.1:8091", "10.10.10.2:8091"},
                        bucket_name = "test",
                        bucketpwd = "test-password",
                    }
    
                    local client, err = couchbase:create_client(conf.hosts, conf.bucket_name, conf.bucketpwd)
                    if client == nil then
                        ngx.log(ngx.ERR, err)
                    end
    
                    -- test set_timeout
                    client:set_timeout(500)
    
                    local key = "test-key"
                    local key1 = "test-key1"
                    -- test set
                    client:set(key, "{}")
                    client:set(key1, "{}")
    
                    -- test get_bluk
                    local values, bluk_err = client:get_bluk(key, key1)
                    if not bluk_err then
                        ngx.say(cjson.encode(values))
                    end
    
                    -- test n1ql
                    local result, query_err = client:query('SELECT country FROM `travel-sample` WHERE name = "Excel Airways";')
                    if not query_err then
                        ngx.say(result)
                    end
    
                    -- test get get_from_replica
                    local value, get_err = client:get(key)
                    if value then
                        ngx.say(value)
                    else
                        if get_err then
                            if string.find(get_err, "Not found") then
                                ngx.log(ngx.INFO, "key not found: ", key, " error: ", get_err)
                                ngx.say(get_from_service())
                            else
                                local value_bak, err_bak = client:get_from_replica(key)
                                if value_bak then
                                    ngx.log(ngx.WARN, "get key from replica success: ", key)
                                    ngx.say(value_bak)
                                else
                                    ngx.log(ngx.ERR, "get replica error: ", key, "error: ", err_bak)
                                    ngx.say(get_from_service())
                                end
                            end
                        end
                    end
    
                    -- test close
                    client:close()
    
                }
            }
        }

Debugging

It is usually convenient to use the lua-cjson library to encode the return values of the couchbase command methods to JSON. For example,

        local cjson = require "cjson"
        ...
        local res, err = client:get("h1234")
        if res then
            print("res: ", cjson.encode(res))
        end

Automatic Error Logging

By default the underlying ngx_lua module does error logging when socket errors happen. If you are already doing proper error handling in your own Lua code, then you are recommended to disable this automatic error logging by turning off ngx_lua's lua_socket_log_errors directive, that is,

        lua_socket_log_errors off;

Check List for Issues

Ensure you configure the connection pool size properly in the set_keepalive . Basically if your NGINX handle n concurrent requests and your NGINX has m workers, then the connection pool size should be configured as n/m. For example, if your NGINX usually handles 1000 concurrent requests and you have 10 NGINX workers, then the connection pool size should be 100.
Ensure you are not using too short timeout setting in the set_timeout or set_timeouts methods. If you have to, try redoing the operation upon timeout and turning off "automatic error logging" (because you are already doing proper error handling in your own Lua code).
If your NGINX worker processes' CPU usage is very high under load, then the NGINX event loop might be blocked by the CPU computation too much. Try sampling a C-land on-CPU Flame Graph and Lua-land on-CPU Flame Graph for a typical NGINX worker process. You can optimize the CPU-bound things according to these Flame Graphs.
If your NGINX worker processes' CPU usage is very low under load, then the NGINX event loop might be blocked by some blocking system calls (like file IO system calls). You can confirm the issue by running the epoll-loop-blocking-distr tool against a typical NGINX worker process. If it is indeed the case, then you can further sample a C-land off-CPU Flame Graph for a NGINX worker process to analyze the actual blockers.

Limitations

This library cannot be used in code contexts like init_by_lua, set_by_lua, log_by_lua*, and header_filter_by_lua* where the ngx_lua cosocket API is not available.

TODO

Bugs and Patches

Please report bugs or submit patches by

creating a ticket on the gitlab.

Author

goecho <gaofeng@qiyi.com>

Copyright and License

This module is licensed under the BSD license.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.