SSL/TLS support for cosocket API?

[Doridian]

So far all I needed to do SSL with has been HTTPS so I settled with using ngx.location.capture and internal proxy locations.
But now I have come accross an issue.
I need to send E-Mail over a server which requires TLS (Amazon SES), which seems undoable at the moment as LuaSec does not work (and prolly would ruin cosocket's asynchronity anyway).
Is there any possible way I could establish communication with the server?

[agentzh]

Hello!

On Sat, Nov 17, 2012 at 4:19 AM, Mark Dietzer notifications@github.com wrote:

So far all I needed to do SSL with has been HTTPS so I settled with using ngx.location.capture and internal proxy locations.
But now I have come accross an issue.
I need to send E-Mail over a server which requires TLS (Amazon SES), which seems undoable at the moment as LuaSec does not work (and prolly would ruin cosocket's asynchronity anyway).
Is there any possible way I could establish communication with the server?

This is still a TODO. I hope I can implement this feature as soon as
possible :) This is my next big item for ngx_lua. I'll keep you posted
:)

Thanks!
-agentzh

What about just providing a Lua interface to the OpenSSL functions in the core (assuming they're there) and then creating the handshaking logic in a resty module like the others?

This it would then be possible to create potentially any kind of encrypted system (I'm planning on creating an SSH one for instance).

[Doridian]

Theres a reason that even Java provides native implementations for SSL/TCP connections.
That reason is pure speed.
You just cannot implement SSL at the same speed in C versus Lua.

Of course. The computationally heavy part would be done in C - using the OpenSSL library (assuming that's the one that's compiled in with the server, which is usually the case).

You'd have a bunch of interfaces like:

ngx.openssl.func()

or even just

openssl.func()

The network logic of dealing with the back and forth communications (which is much less computationally heavy) would be done using Lua/cosockets, like it is in resty_memcached, resty_mysql etc.

[Doridian]

I honestly wish resty_mysql and especially resty_redis were C modules and not Lua ones...
Would manage to get out a few more percent I am sure.
But that's a different topic.

[bakins]

I honestly wish resty_mysql and especially resty_redis were C modules and not Lua ones...
Would manage to get out a few more percent I am sure.

Not really. Luajit is very, very fast. The overhead of "switching" between Lua and C is often more than any gains. Actually read the implementation of the mysql and redis clients. They are mostly string manipulation and Luajit is often faster than C at this.

If they were in pure C, then they may be a bit faster (see Brian's commen), but they may not, and there are many more issues and potential sources of bugs with creating them in C.

What ngx.cosocket provides, though, is a "safe" platform from a networking/socket level (which is usually the most difficult part of it to get right), and then just leaves it up to the Lua code to deal with the text-parsing side of things (which is generally much, much easier, and more programmers are comfortable doing it).

This makes the platform much more developer-friendly, and provides a greater speed of development. To show this in real terms, just look at how many modules have been created to connect to back-ends since it was first released (e.g. Redis, Mongo, HTTP, MySQL, Memcached). They all popped up in a matter of months (I know agentzh/chaoslawful wrote several of them, but the dev speed is much higher using this method).

By providing a full OpenSSL interface, it would similarly be possible to create SSL, TLS, HTTPS, SSH, SFTP, FTPS (and any others that use encryption). Sure the implementation might be very slightly slower than a completely native TSL implementation, but it would be much more flexible and the development of libraries that use encryption would probably happen much quicker than if the only implementations were native ones.

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 3:01 PM, simpl.it notifications@github.com wrote:

What about just providing a Lua interface to the OpenSSL functions in the core (assuming they're there) and then creating the handshaking logic in a resty module like the others?

We've already started doing that in the lua-resty-string library:

https://github.com/agentzh/lua-resty-string

We're exposing quite a few OpenSSL's C API via LuaJIT 2.0's FFI mechanism :)

But it may still worthwhile to just expose the existing SSL C API in
the Nginx core to the Lua land. I'm not 100% sure right now :)

This it would then be possible to create potentially any kind of encrypted system (I'm planning on creating an SSH one for instance).

Agreed. I've also been thinking about creating the nonblocking
lua-resty-ssh client library for SSH servers :D

Thanks!
-agentzh

I think FFI might be a good way to get things started on that. It has some very interesting possibilites for relatively easy integration of other third-party libraries, too. Have you used it for anything much, yet?

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 5:53 PM, simpl.it notifications@github.com wrote:

I think FFI might be a good way to get things started on that. It has some
very interesting possibilites for relatively easy integration of other
third-party libraries, too. Have you used it for anything much, yet?

Until now, just in lua-resty-string :)

But I'm going to port all of the existing Nginx API for Lua provided by
ngx_lua over to FFI very soon, in order to take full advantage of the
LuaJIT 2.0's JIT compiler (because CFunction calls will abort the traces
and make LuaJIT fall back to the interpreter mode).

When the FFI move in ngx_lua happens, I think I'd preserve the old
CFunction-based API to keep compatibility with the standard Lua 5.1.x
interpreters. I may use C macros to toggle these two separate code paths in
ngx_lua.

Best regards,
-agentzh

If I were you, I'd consider splitting them into two separate branches. Converting everything to FFI is a pretty major change, and considering the benefits of JIT, I think it would be less headache in the long-run to make a clean break.

Both versions can be supported, but the focus can be on the JIT-only implementation and the C-only implementation could be considered a legacy version.

I think doing it this way would be less of a headache in the long-run.

I can envisage the importing of a large number of third-party C libraries using FFI, and there could even be an organized platform for loading ngx_lua-compatible modules (perhaps a bit like an ngx_lua equivalent to luarocks - ngxluarocks?).

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 7:05 PM, simpl.it notifications@github.com wrote:

If I were you, I'd consider splitting them into two separate branches. Converting everything to FFI is a pretty major change, and considering the benefits of JIT, I think it would be less headache in the long-run to make a clean break.

Both versions can be supported, but the focus can be on the JIT-only implementation and the C-only implementation could be considered a legacy version.

I think doing it this way would be less of a headache in the long-run.

I must say it makes a lot of sense for future development :) I surely
don't want to write two versions of implementation everytime I try to
add something new to ngx_lua :)

Forking the code base could be a better approach and I'll think more
about it :) Before it really happens, I'll experiment the FFI move in
a git branch first :)

I can envisage the importing of a large number of third-party C libraries using FFI, and there could even be an organized platform for loading ngx_lua-compatible modules (perhaps a bit like an ngx_lua equivalent to luarocks - ngxluarocks?).

I'm glad that we're thinking along the same line ;) I've indeed been
thinking about launching a luaresty.org site hosting ngx_lua
compatible Lua libraries, similar to CPAN and LuaRocks :)

Best regards,
-agentzh

luaresty.org sounds like a good idea. It might be worth considering organizing everything around openresty.org, too. I think if everything's in one place, it might make things easier for (some) people.

Wrt the module system, you may want to think about whether it is worth considering adding DSO support to the lua core, so that Lua modules that use FFI could potentially load the shared library on the fly (or at least as part of lua_init). I think when you're considering Nginx largely as a webserver with a few 'bonus' features, it's probably just as well to not include DSO, but if you're envisaging ngx_lua to being a complete replacement for PHP/Python etc (which I am long-term), then I think it makes more sense to not require that all objects be compiled (or at least linked) from the beginning.

It also may be worth considering a interface for FFI-loaded modules that provide may assist with e.g. memory management with libraries that allow the passing of one or several memory-allocation functions (e.g. by saving things like a pointer to the current request to a global variable that any FFI-loaded module could access).

[agentzh]

Hello!

On Wed, Nov 21, 2012 at 8:00 PM, simpl.it notifications@github.com wrote:

luaresty.org sounds like a good idea. It might be worth considering organizing everything around openresty.org, too. I think if everything's in one place, it might make things easier for (some) people.

Maybe. We'll see :)

Wrt the module system, you may want to think about whether it is worth considering adding DSO support to the lua core, so that Lua modules that use FFI could potentially load the shared library on the fly (or at least as part of lua_init). I think when you're considering Nginx largely as a webserver with a few 'bonus' features, it's probably just as well to not include DSO, but if you're envisaging ngx_lua to being a complete replacement for PHP/Python etc (which I am long-term), then I think it makes more sense to not require that all objects be compiled (or at least linked) from the beginning.

The LuaJIT FFI can already load .so on-the-fly. Please check out the
ffi.load() API:

http://luajit.org/ext_ffi_api.html

Every Lua module can just look for .so/.dylib/.dll files in the search
paths specified by package.cpath and load it via ffi.load to the Lua
land on demand ;)

It also may be worth considering a interface for FFI-loaded modules that provide may assist with e.g. memory management with libraries that allow the passing of one or several memory-allocation functions (e.g. by saving things like a pointer to the current request to a global variable that any FFI-loaded module could access).

Maybe.

But accessing the raw ngx_http_request_t struct from within FFI
directly is a bad idea because the memory layout of ngx_http_request_t
may change upon different ./configure options (like the presence of
libssl, libz, or libpcre). We'll have to think more about it.

For the FFI move in the ngx_lua core, I'm going to wrap the Nginx C
APIs with an intermediate C function layer so that we don't have to
directly manipulate the underlying Nginx C structures (like the
troublesome ngx_http_request_t struct) from within Lua via FFI at all.
I'll work out all these details very soon :)

Best regards,
-agentzh

The LuaJIT FFI can already load .so on-the-fly.

Cool. No need to add anything, then. :-)

But accessing the raw ngx_http_request_t struct from within FFI
directly is a bad idea because the memory layout of ngx_http_request_t
may change upon different ./configure options (like the presence of
libssl, libz, or libpcre). We'll have to think more about it.

What I was thinking about specifically was providing a pointer to the request pool. It would be useful for libraries that allow the defining of external memory allocators (which usually require a pointer to some custom struct).

Wrt providing direct access to the http_request_t struct, it would certanly be possible to create a lua interface with the same struct that was compiled into the server, but would require some extra ./configure time code to generate the correct struct, and then load it as a single interface.

That said... I think wrapping the FFI stuff around an intermediate layer may well be better idea. I can see all sorts of problems starting if Lua has complete access to r*.