Working with cookies in a Nginx module

Posted by vkholodkov

PPC advertising campaign via nginx moduleImagine you run a PPC advertising campaign and you want to find out how many visitors coming from a search engine result in sales. We will create an Nginx module and use cookies for this purpose. Whenever a visitor clicks on your ad, a landing page is requested with a tracking argument in it. The tracking argument looks  like that: ‘?source=whatever’. We will put the content of tracking argument into a cookie that will be called a source cookie and write it into a log file. Whenever a visitor makes a transaction (e.g. buys an article or makes a booking), the name of the source will be recorded and we will be able to easily attribute every transaction to a source.

Let’s start with declaring a structure that will contain configuration of our module:

typedef struct {
    ngx_str_t                       name;
    time_t                          expires_time;
    ngx_str_t                       domain;
    ngx_str_t                       path;
    ngx_http_complex_value_t        *value;
} ngx_http_source_cookie_loc_conf_t;

We need to store the name of the cookie, the expiration time in seconds, the domain and the path. Should  the cookie name in certain location be empty, the module will not be activated. We also need to store a complex value that will determine the content of the cookie. A complex value is a pre-compiled byte code that can be executed at runtime in order to get a string value. The complex value is generated at configuration time from a string with place holders that is called a script:

    ngx_http_compile_complex_value_t ccv;
    sclc->value = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t));
    if(sclc->value == NULL) {
        return NGX_CONF_ERROR;
    }
    ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));
    ccv.cf = cf;
    ccv.value = &value[2];
    ccv.complex_value = sclc->value;
    if(ngx_http_compile_complex_value(&ccv) != NGX_OK) {
        return NGX_CONF_ERROR;
    }

Here we take a script from the second argument of the directive value[2] and compile it into the complex value sclc->value. In the configuration file we will put a script “$arg_source” into this argument and this script will evaluate into the value of URI argument source at runtime. Scripts give you flexibility in case if you want to change the name of the tracking argument or if you want to start using something else for tracking.

Next, let’s create a function that will find and parse the source cookie:

static ngx_int_t
ngx_http_source_cookie_parse(ngx_http_request_t *r, ngx_str_t *value) {
    ngx_int_t                          rc;
    ngx_http_source_cookie_loc_conf_t  *sclc;
    sclc = ngx_http_get_module_loc_conf(r, ngx_http_source_cookie_module);
    /*
     * Find the cookie
     */
    rc = ngx_http_parse_multi_header_lines(&r->headers_in.cookies,
        &sclc->name, value);
    if(rc == NGX_DECLINED) {
        return rc;
    }
    return NGX_OK;
}

Here we use the function ngx_http_parse_multi_header_lines to locate a header line that contains the cookie with the name that is specified by the configuration in the list of cookies r->headers_in.cookies. This function returns the index of the element that contains a cookie with specified name or NGX_DECLINED if no such cookie is found. Since we are only interested in the presence of the cookie, the function ngx_http_source_cookie_parse simply returns NGX_OK if the cookie is found. You can implement some validation/post-processing here if you want.

We also need a function that takes the value of the specified argument and sets a cookie with that value. First, we will evaluate the complex value from the configuration of the module into a string:

    ngx_str_t         value;
    if(conf->value == NULL) {
        return NGX_OK;
    }
    if(ngx_http_complex_value(r, conf->value, &value) != NGX_OK) {
        return NGX_ERROR;
    }
    if(value.len == 0) {
        return NGX_OK;
    }

Here the function ngx_http_complex_value takes a pointer to the complex value conf->value, evaluates it and writes a result into a string value. Whenever the complex value is not present or evaluates to an empty string, we will avoid setting the cookie. After we’ve finished with evaluating the source, we can proceed to setting the cookie. To set a cookie we just need to add a “Set-Cookie” header line to the list of response header lines:

    expires = ngx_time() + conf->expires_time;
    len = conf->name.len + 1 + value.len;
    if(conf->expires_time != 0) {
        len += sizeof("; expires=") - 1 +
            sizeof("Mon, 01 Sep 00:00:00 GMT") - 1;
    }
    if(conf->domain.len) {
        len += conf->domain.len;
    }
    if(conf->path.len) {
        len += conf->path.len;
    }
    cookie = ngx_pnalloc(r->pool, len);
    if (cookie == NULL) {
        return NGX_ERROR;
    }
    p = ngx_copy(cookie, conf->name.data, conf->name.len);
    *p++ = '=';
    p = ngx_copy(p, value.data, value.len);
    if(conf->expires_time != 0) {
        p = ngx_cpymem(p, "; expires=", sizeof("; expires=") - 1);
        p = ngx_http_cookie_time(p, expires);
    }
    p = ngx_copy(p, conf->domain.data, conf->domain.len);
    p = ngx_copy(p, conf->path.data, conf->path.len);
    set_cookie = ngx_list_push(&r->headers_out.headers);
    if (set_cookie == NULL) {
        return NGX_ERROR;
    }
    set_cookie->hash = 1;
    ngx_str_set(&set_cookie->key, "Set-Cookie");
    set_cookie->value.len = p - cookie;
    set_cookie->value.data = cookie;
    ngx_log_debug1(NGX_LOG_DEBUG_HTTP, r->connection->log, 0,
                   "set cookie: \"%V\"", &set_cookie->value);

Norway Spruce trees covered with snowHere we calculate the amount of the memory that we need for storing the content of the cookie including the path, the domain and the expiration time parameters, allocate this amount of memory from the pool of the request and fill in the content of the cookie. We use function ngx_http_cookie_time to convert time stamp into RFC time stamp.

Note that if our module is used in location that is proxied to another web server, the response header will be replaced by the header from a proxied web server. Since most likely it will be the case, the best place to set the cookie is in a response header filter. In the header filter we’ll check if the browser has already sent us our cookie and if not, we will set it:

static ngx_int_t
ngx_http_source_cookie_header_filter(ngx_http_request_t *r)
{
    ngx_int_t                           rc;
    ngx_http_source_cookie_loc_conf_t   *sclc;
    ngx_str_t                           value;
    sclc = ngx_http_get_module_loc_conf(r, ngx_http_source_cookie_module);
    if((sclc->name.len == 0)
        || r != r->main
        || r->internal
        || r->error_page
        || r->post_action)
    {
        return ngx_http_next_header_filter(r);
    }
    rc = ngx_http_source_cookie_parse(r, &value);
    if(rc == NGX_OK) {
        return ngx_http_next_header_filter(r);
    }
    rc = ngx_http_source_cookie_set(r, sclc);
    if(rc != NGX_OK) {
        return NGX_ERROR;
    }
    return ngx_http_next_header_filter(r);
}

Note that we avoid setting the cookie in subrequests, internal locations, error pages, post actions or in locations where the cookie name is not defined, i.e. our module is not enabled. We also don’t set the cookie if it is already set.

Next, we need to work on code that processes the configuration of our module. Let’s create definition of configuration directives for our module:

static char *ngx_http_source_cookie_command(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
static ngx_command_t ngx_http_source_cookie_commands[] = {
    { ngx_string("source_cookie"),
      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF|NGX_CONF_TAKE12,
      ngx_http_source_cookie_command,
      NGX_HTTP_LOC_CONF_OFFSET,
      0,
      NULL },
    { ngx_string("source_cookie_domain"),
      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF|NGX_CONF_TAKE1,
      ngx_conf_set_str_slot,
      NGX_HTTP_LOC_CONF_OFFSET,
      offsetof(ngx_http_source_cookie_loc_conf_t, domain),
      NULL },
    { ngx_string("source_cookie_path"),
      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF|NGX_CONF_TAKE1,
      ngx_conf_set_str_slot,
      NGX_HTTP_LOC_CONF_OFFSET,
      offsetof(ngx_http_source_cookie_loc_conf_t, path),
      NULL },
    { ngx_string("source_cookie_expires"),
      NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF|NGX_CONF_TAKE1,
      ngx_conf_set_sec_slot,
      NGX_HTTP_LOC_CONF_OFFSET,
      offsetof(ngx_http_source_cookie_loc_conf_t, expires_time),
      NULL },
      ngx_null_command
};

A directive source_cookie will specify the name and the content of the cookie in its arguments and it will determine locations where the module will be active. 3 auxiliary directives source_cookie_domain, source_cookie_path and source_cookie_expires will specify the domain,  the path and the expiration time of the cookie respectively. For these 3 directive we will use standard configuration handlers ngx_conf_set_str_slot and ngx_conf_set_sec_slot. For the directive source_cookie we will create a custom handler:

static char *
ngx_http_source_cookie_command(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
    ngx_http_source_cookie_loc_conf_t *sclc = conf;
    ngx_str_t                         *value;
    ngx_http_compile_complex_value_t  ccv;
    if(sclc->name.data != NULL) {
        return "is duplicate";
    }
    value = cf->args->elts;
    sclc->name = value[1];
    if(cf->args->nelts > 2) {
        sclc->value = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t));
        if(sclc->value == NULL) {
            return NGX_CONF_ERROR;
        }
        ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));
        ccv.cf = cf;
        ccv.value = &value[2];
        ccv.complex_value = sclc->value;
        if(ngx_http_compile_complex_value(&ccv) != NGX_OK) {
            return NGX_CONF_ERROR;
        }
    }
    return NGX_CONF_OK;
}

This function includes the fragment of code above. The logic is simple: initialise the cookie name from the first argument and compile a complex value from the second argument if there is any.

Now there is just a bit of boring stuff left: the creating and merging of the configuration:

static void *
ngx_http_source_cookie_create_loc_conf(ngx_conf_t *cf)
{
    ngx_http_source_cookie_loc_conf_t  *conf;
    conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_source_cookie_loc_conf_t));
    if(conf == NULL) {
        return NGX_CONF_ERROR;
    }
    conf->expires_time = NGX_CONF_UNSET;
    conf->value = NGX_CONF_UNSET_PTR;
    return conf;
}
static char *
ngx_http_source_cookie_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child)
{
    ngx_http_source_cookie_loc_conf_t *prev = parent;
    ngx_http_source_cookie_loc_conf_t *conf = child;
    ngx_conf_merge_str_value(conf->name, prev->name, "");
    ngx_conf_merge_value(conf->expires_time, prev->expires_time, 0);
    ngx_conf_merge_str_value(conf->domain, prev->domain, "");
    ngx_conf_merge_str_value(conf->path, prev->path, "");
    ngx_conf_merge_ptr_value(conf->value, prev->value, NULL);
    return NGX_CONF_OK;
}

The function ngx_http_source_cookie_create_loc_conf is called whenever Nginx is about to create a configuration for our module for a certain location. We allocate memory for our configuration structure and initialise expires_time and value. The other members are strings and they will be set to (ngx_str_t){ NULL, 0 } by means of the function ngx_pcalloc.

The function ngx_http_source_cookie_merge_loc_conf is called whenever Nginx needs to assign default values to members of the configuration structure or inherit them from a less specific configuration, e.g. from the configuration of a location to the configuration of a limit_except block. In our implementation we basically inherit everything or assign default values whenever there is nothing to inherit, that is, an empty string to all string members, 0 to expiration time and a NULL value to cookie value.

Finally, we need code that puts  everything together: module definition and post-initialisation handler:

static ngx_http_module_t ngx_http_source_cookie_module_ctx = {
    NULL,                             /* preconfiguration */
    ngx_http_source_cookie_init,             /* postconfiguration */
    NULL,                             /* create main configuration */
    NULL,                             /* init main configuration */
    NULL,                             /* create server configuration */
    NULL,                             /* merge server configuration */
    ngx_http_source_cookie_create_loc_conf,  /* create location configuration */
    ngx_http_source_cookie_merge_loc_conf    /* merge location configuration */
};
ngx_module_t ngx_http_source_cookie_module = {
    NGX_MODULE_V1,
    &ngx_http_source_cookie_module_ctx,      /* module context */
    ngx_http_source_cookie_commands,         /* module directives */
    NGX_HTTP_MODULE,                  /* module type */
    NULL,                             /* init master */
    NULL,                             /* init module */
    NULL,                             /* init process */
    NULL,                             /* init thread */
    NULL,                             /* exit thread */
    NULL,                             /* exit process */
    NULL,                             /* exit master */
    NGX_MODULE_V1_PADDING
};
static ngx_http_output_header_filter_pt ngx_http_next_header_filter;
static ngx_int_t
ngx_http_source_cookie_init(ngx_conf_t *cf)
{
    ngx_http_next_header_filter = ngx_http_top_header_filter;
    ngx_http_top_header_filter = ngx_http_source_cookie_header_filter;
    return NGX_OK;
}

The HTTP module definition will provide Nginx with a reference to the configuration initialisation and merging handlers as well as a post-initialisation handler. The post-initialisation handler on its turn, will add our header filter to the chain of the header filters.

Here is an example configuration for our module:

events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  with_source  '$remote_addr - $remote_user [$time_local] $status '
            '"$request" $body_bytes_sent "$http_referer" '
            '"$http_user_agent" "$host" $gzip_ratio "$cookie_source"';
    server {
        listen       80;
        server_name  localhost;
        access_log logs/access.log with_source;
        root html;
        source_cookie source "$arg_source";
        source_cookie_domain localhost;
        source_cookie_expires 7d;
    }
}

This configuration makes Nginx set the cookie whenever the URI argument source is not empty and causes it to write the content of the source cookie into the access log. We use the standard Nginx variable $cookie_source to get the content of our cookie. The expiration time of the cookie is set to 7 days.

This is how a visitor session could look like in a log file:

visitor session could look like in a log file

It is simple to modify this module, so that it removes the cookie after a transaction has been completed by setting the expiration time to current timestamp. This is necessary for attributing subsequent transactions to correct source. Some efforts will be required to implement validation of the cookie in order to prevent bots and competitors from influencing your results.

Here you can find an archive with the full source code of the module.

This entry was posted in nginx. Bookmark the permalink.