Category: howto

Nginx Plus Docker Image + lua-resty-openidc for OAuth termination

I need a Docker image with Nginx Plus and configured lua-resty-openidc to use Keycloak OAuth provider.
I made it based on this article Deploying NGINX and NGINX Plus with Docker but there was few additional non trivial steps so here is my result.
Create a folder with nginx plus repo keys (nginx-repo.crt and nginx-repo.key)
Then create a Dockerfile with the following content:

FROM ubuntu:artful

# Download certificate and key from the customer portal (https://cs.nginx.com)
# and copy to the build context
COPY nginx-repo.crt /etc/ssl/nginx/
COPY nginx-repo.key /etc/ssl/nginx/

# Install NGINX Plus
RUN set -x \
  && apt-get update && apt-get upgrade -y \
  && apt-get install --no-install-recommends --no-install-suggests -y apt-transport-https ca-certificates \
  && apt-get install -y lsb-release wget \
  && wget http://nginx.org/keys/nginx_signing.key && apt-key add nginx_signing.key \
  && wget -q -O /etc/apt/apt.conf.d/90nginx https://cs.nginx.com/static/files/90nginx \
  && printf "deb https://plus-pkgs.nginx.com/ubuntu `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-plus.list \
  && apt-get update && apt-get install -y nginx-plus nginx-plus-module-lua nginx-plus-module-ndk luarocks libssl1.0-dev git

RUN set -x \
  && apt-get remove --purge --auto-remove -y \
  && rm -rf /var/lib/apt/lists/*

# Forward request logs to Docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
  && ln -sf /dev/stderr /var/log/nginx/error.log

RUN luarocks install lua-resty-openidc
RUN luarocks install lua-cjson
RUN luarocks install lua-resty-string
RUN luarocks install lua-resty-http
RUN luarocks install lua-resty-session
RUN luarocks install lua-resty-jwt

EXPOSE 80

STOPSIGNAL SIGTERM

CMD ["nginx", "-g", "daemon off;"]

Here you can see that we installing not only nginx-plus but also nginx-plus-module-lua and nginx-plus-module-ndk modules which are needed to run lua-resty-openidc.
Since open lua-resty-openidc is distributed via luarocks package manager we need to install it too and then install all needed packages via luarocks. For lua-crypto dependency you need to install libssl1.0-dev package with OpenSSL headers and for some other package we needed git, don't ask me why, I have no idea.
FYI: openidc is installed into file /usr/local/share/lua/5.1/resty/openidc.lua

Then you need to build an image with

docker build --no-cache -t nginxplus .

If you have a Docker Registry inside your company you can publish the image there:

docker tag nginxplus your.docker.registry:5000/nginxplus
docker push your.docker.registry:5000/nginxplus

Not you have an image and you can run it. All you need is to mount you server config into /etc/nginx folder. Consider you have a docker-compose.yml file with the following content:

version: '3'
services:
  gw-nginx:
    image: your.docker.registry:5000/nginxplus
    container_name: gw-nginx
    volumes:
      - ~/gateway/etc/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ~/gateway/etc/nginx/conf.d/:/etc/nginx/conf.d/
      - /etc/localtime:/etc/localtime
    ports:
      - 80:80
  gw-keycloak:
    image: jboss/keycloak
    container_name: gw-keycloak
    volumes:
      - /etc/localtime:/etc/localtime
    ports:
      - 8080:8080
    environment:
      - KEYCLOAK_USER=root
      - KEYCLOAK_PASSWORD=changeMePlease
      - PROXY_ADDRESS_FORWARDING=true

Now create a gateway folder:
mkdir ~/gateway
cd ~/gateway
And plase nginx.conf file into ~/gateway/etc/nginx/nginx.conf. The most important is to place this two lines:

...
http {
  resolver yourdnsip;
  lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
  lua_ssl_verify_depth 5;
  ...
}

For some reason without this lines Lua scripts can’t resolve upstreams.

Create ~/gateway/etc/nginx/conf.d/default.conf file and configure it as descibed in restly-openidc documentation.
Finally you can run it with docker-compose up -d command.

Реклама
Letsencrypt

How to: LetsEncrypt HTTPS on OpenWRT with uhttpd

My old router TP Link WRN740N hosting my homepage stokito.name and it’s too small to handle full LetsEncrypt certbot installer and OpenSSL. So if you want to enable HTTPS you have to run certbot on some other machine and then upload to router.
Here I would like to show how I did that.

Manual installation

The fisrt step is to use manual certs installation from my laptop and renew them after 3 month. Actually this can be automated too latter.

Now, lets generate certs for your domain:

$ sudo certbot certonly --manual --preferred-challenges http

Answer all the questions and it will ask you to upload a file to your router:

-------------------------------------------------------------------------------
Create a file containing just this data:

1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KuC8g.OKKBaAC2SgfXHQyvgKrLkn3zyCNH82xHgKsMg9OQQJE

And make it available on your web server at this URL:

http://www.stokito.name/.well-known/acme-challenge/1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KuC8g

-------------------------------------------------------------------------------
Press Enter to Continue

You need to create the folder on router:

# mkdir -p ./.well-known/acme-challenge

Then upload the files via SCP from your computer to router:

$ echo "1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g.OKKBaAC2SgfXHQyvgKrLkn3zyCNH82xHgKsMg9OQQJE" > 1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g

$ scp ./1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g root@192.168.1.1:/www/.well-known/acme-challenge/1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g

BTW, there is no any analogue of WinSCP for Linux but you can try to run it on Wine.

Then go back to certbot and press Enter. It will check that files are in place and accessible from web.

 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/stokito.name/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/stokito.name/privkey.pem
   Your cert will expire on 2018-01-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again.

It generated two files: privkey.pem and fullchain.pem (which is not public key!)

Finally, you need to convert the private key and the certificate from the ASCII-armored PEM format to the more economical binary DER format used by uhttpd:

openssl rsa -in privkey.pem -outform DER -out uhttpd.key
openssl x509 -in fullchain.pem -outform DER -out uhttpd.crt

Upload them to the router

$ scp uhttpd.crt root@192.168.1.1:/etc/uhttpd.crt
$ scp uhttpd.key root@192.168.1.1:/etc/uhttpd.key

On the router you need to install uhttpd-mod-tls:

# opkg update
# opkg install uhttpd-mod-tls

edit /etc/config/uhttpd as described in docs i.e. like this

config uhttpd 'stokito.name'
    list listen_http '31.172.137.103:80'
    list listen_https '31.172.137.103:443'
    option redirect_https '1'
    option home '/www'
    option rfc1918_filter '0'
    option cert '/etc/uhttpd.crt'
    option key '/etc/uhttpd.key'

Here 31.172.137.103 is my public static IP.
Note that 443 port should be opened in /etc/config/firewall:

config rule
    option target 'ACCEPT'
    option src 'wan'
    option proto 'tcp'
    option dest_port '443'
    option name 'HTTPS'

Then restart the firewall and uhttpd server:

# /etc/init.d/firewall restart
# /etc/init.d/uhttpd restart

Now try your site in browser. But please check your site latter: I noticed that uhttpd was down but after restart it worked well.

Renewing cert

… So it passed 3 months and my cert got expired and I need to renew it. It’s funny that today is an Old New Year and from window I hear some concert on my street.
This time I decided not to use manual mode and use standalone mode instead: certbot starts itself an https server on 443 port and I need to shut down webserver on my router and enable 443 port forwarding from router to my laptop.

So let’s do that:
1. Connect to router and stop uhttpd service:

$ ssh root@192.168.1.1
# /etc/init.d/uhttpd stop
  1. Enable 443 port forwarding. Download firewall config from router:
$ scp root@192.168.1.1:/etc/config/firewall ./
  1. Edit and comment out current rule for 443 port. If you used that one that I mentioned before then:
# temporarry comment out the rule
#config rule
#  option target 'ACCEPT'
#  option src 'wan'
#  option proto 'tcp'
#  option dest_port '443'
#  option name 'HTTPS'

Then add a HTTPS forwarding rule:

config 'redirect'
    option 'name' 'HTTPS_to_laptop'
    option 'src' 'wan'
    option 'proto' 'tcp'
    option 'src_dport' '443'
    option 'dest_ip' '192.168.1.144'
    option 'dest_port' '443'
    option 'target' 'DNAT'
    option 'dest' 'lan'

Where 192.168.1.144 is the ip of the your laptop. Run ifconfig to see it.

  1. Upload the new firewall config to router:
$ scp ./firewall root@192.168.1.1:/etc/config/
  1. Now restart firewall service on router:
# /etc/init.d/firewall restart
  1. Now your laptop’s 443 port is exposed to the world. So lets tun certbot:
$ sudo certbot certonly --standalone --preferred-challenges tls-sni --cert-name=stokito.name
  1. Then convert the keys do DER format and upload to router as was described above. Then disable forwardind firewall rule and rollback previous and restart firewall and uhttpd.

Now you certs was renewed.

Transliteration to ASCII

If you need to make a translitartion from any language to ASCII symbols you can use a Transliterator from ICU4J.

private static final String TRANSLITERATION_RULE = "Any-Latin; Latin-ASCII";

private static String transliterate(String name) {
    String ascii = TRANSLITERATOR.transliterate(name);
    // Some Russian names may contain Soft Sign ( Ь ) and ( Ъ ) that may cause error http://sourceforge.net/p/icu/mailman/message/34413588/
    ascii = ascii.replaceAll("[ʹʺ]", "");
    return ascii;
}

ICU Transform Demonstration
1) Select «Names» from «Inset sample» combo box.
2) Insert the rule «Any-Latin; Latin-ASCII» to the «Compound 1» fields.
3) Press «Transform» button

Also a good example:
How do I convert Chinese characters to their Latin equivalents?

What are the system Transliterators available with ICU4J?

[HOWTO] WYSIWYG with Grails + CKEditor + Spring Security

Add last version of Grails CKEditor plugin to BuildConfig.groovy

grails.project.dependency.resolution = {
...
    plugins {
        ...
        compile ":ckeditor:4.4.1.0"
    }
}

Plugin version corresponding to bundled CKEditor version, and may be outdated.
To update it, and you use assets pipeline plugin, download CKEditor and unpack to grails-app/assets/ckeditor.
Then you should add dependency to application.js like this

//= require jquery-1.11.1.min
//= require jquery-ui.min
//= require_tree plugins
//= require_tree globalize
//= require bootstrap
//= require ../ckeditor/ckeditor
//= require_self

If CKEditor version is good enough for you, just put ckeditor:resources tag into head of view:

<html>
<head>
    <title>Example</title>
    <ckeditor:resources/>
</head>

Then you can insert CKEditor in form with tag ckeditor:editor. For example editor of article content:

<ckeditor:editor name="content" height="400px" width="80%" userSpace="${currentUser.id}">${article.content}</ckeditor:editor>

Here used attribute userSpace with value of current user's id. It makes all user images upload to server separated to their own folders.

But you should also restrict access to users spaces. If you use Spring Security (S2) plugin you can create custom filter with command grails create-filters OfmSecurity:

class OfmSecurityFilters {
 def springSecurityService
 def filters = {
 all(uri: '/ck/ofm/**') {
 before = {
 if (springSecurityService.currentUser?.id != params.space?.toLong()) {
 redirect(controller: "user", action: "login")
 return false
 }

 }
 after = { Map model ->

 }
 afterView = { Exception e ->

 }
 }
 }
}

[Чтиво] Три очень классные практические статьи-инструкции для программистов

Периодически почитываю что накопилось в закладках. Вот отличные прагматичные статьи вкурив которые сразу получите +2 к экспе девелопера.
Экстремальное программирование: Pair Programming чёткая инструкция по делу что такое парное программирование как его делать на практике и самое главное как его НЕ делать.

Парное программирование в аутсорсинге: достижение взаимопонимания с техническими специалистами заказчика Классная статья которую обязан прочесть каждый программист-аутсорсер. Вывод: если у вас есть часть удалённой команды то парно педалить код вы просто обязаны.

Практика рефакторинга в больших проектах в сотый раз о вечном. Для тех кто ещё халтурит и не прочёл замечательную книгу Working Effectively with Legacy Code

[Grails] Mock MySQL database in test environment

Grails supports different data sources configured per environment.
If you use MySQL database in production and want to write integration tests you may prefer H2 database because it simpler.
For example it can automatically create in-memory database from the domain classes of GORM model.
And after running tests it will drop this database when application will shutting down.
H2 allows you to use full set of SQL commands, but they may differ from MySQL.
To imitate MySQL behavior you need to add options Mode=MySQL;DATABASE_TO_UPPER=FALSE;IGNORECASE=TRUE to connection string.

From documentation:

MySQL Compatibility Mode

To use the MySQL mode, use the database URL jdbc:h2:~/test;MODE=MySQL or the SQL statement SET MODE MySQL.

  • When inserting data, if a column is defined to be NOT NULL and NULL is inserted, then a 0 (or empty string, or the current timestamp for timestamp columns) value is used. Usually, this operation is not allowed and an exception is thrown.
  • Creating indexes in the CREATE TABLE statement is allowed using INDEX(..) or KEY(..). Example: create table test(id int primary key, name varchar(255), key idx_name(name));
  • Meta data calls return identifiers in lower case.
  • When converting a floating point number to an integer, the fractional digits are not truncated, but the value is rounded.
  • Concatenating NULL with another value results in the other value.

Text comparison in MySQL is case insensitive by default, while in H2 it is case sensitive (as in most other databases). H2 does support case insensitive text comparison, but it needs to be set separately, using SET IGNORECASE TRUE. This affects comparison using =, LIKE, REGEXP.

But raw SQL may not work, because MySQL field names are case sensitive, GORM expect them lowercased, but by default H2 creates them uppercased.
To fix that you need to set property DATABASE_TO_UPPER to FALSE

Database setting DATABASE_TO_UPPER (default: true).
Database short names are converted to uppercase for the DATABASE() function, and in the CATALOG column of all database meta data methods. Setting this to «false» is experimental. When set to false, all identifier names (table names, column names) are case sensitive (except aggregate, built-in functions, data types, and keywords).

So final Config.groovy will looks like:

environments {
    test {
        dataSource {
            dbCreate = 'create'
            url = 'jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;MODE=MySQL;DATABASE_TO_UPPER=FALSE;IGNORECASE=TRUE'
            pooled = true
            driverClassName = org.h2.Driver.class.canonicalName
            dialect = H2Dialect
            username = 'sa'
            password = ''
            logSql = true
            formatSql = true
            pooled = true
        }
    }
}

[Grails] Избегайте использования Environment вне файлов конфигураций

Внутри Grails есть великолепный механизм для условного выполнения кода в зависимости от текущей среды (Environment).
Например внутри DataSource.groovy можно указывать различные настройки базы данных:

// environment specific settings
environments {
    development {
        dataSource {
            dbCreate = "create-drop"
            url = "jdbc:h2:mem:devDb;MVCC=TRUE;LOCK_TIMEOUT=10000"
        }
    }
    production {
        dataSource {
            dbCreate = "update"
            url = "jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000"
        }
    }
}

Наверняка у вас почти в каждом файле конфигурации есть настройки под среду.

Но очень часто я натыкался на то что класс Environment.current начинает использоваться внутри контроллеров, вьюх и сервисов. А стандартный тег if и вовсе имеет атрибут для проверки текущего окружения:

<g:if env="test"> ... </g:if>

Я постепенно пришёл к тому что этого следует избегать, потому что теряется читаемость и гибкость. Вместо этого лучше явно создать опцию в Config.groovy, включать или выключать её в зависимости от среды а и потом проверять её. Вот например что делает этот код?

    <g:if env="test">
        <meta name="controller" content="${controllerName}"/>
        <meta name="action" content="${actionName}"/>
    </g:if>

На самом деле этот код добавляет имя контроллера и экшена которые отрисовли страницу и они потом проверяются функциональным тестом. Это это бывает очень удобно для дебага.
А теперь давайте мы создадим в файле Config.groovy опцию

environments {
    development {
        com.example.showActionNameInPageMeta = true
    }
    test {
        com.example.showActionNameInPageMeta = true
    }
    production {
        com.example.showActionNameInPageMeta = false
    }
}

и теперь

    <g:if test="${grailsApplication.config.com.example.showActionNameInPageMeta}">
        <meta name="controller" content="${controllerName}"/>
        <meta name="action" content="${actionName}"/>
    </g:if>

В результате эта опция улучшила читаемость кода. Кроме того теперь мы всегда можем добавить новое окружение, например testFunctional для функциональных тестов и внутри включить нашу опцию, при этом не переписывая код.
И что особенно важно, что в случае каких то проблем в продакшене сисадмин может поменять конфигурацию и перезапустить сервер не перекомпилируя приложение.
Кроме того, все такие опции теперь не размазаны по коду а лежат в одном месте где их все можно посмотреть.

В более широком смысле такой подход называется Feature flag, и его активно используют например в Amazon.