Category: howto

Setup DDNS/DynDNS in OpenWrt

I serving my small homepage stokito.com directly from my router with OpenWrt Linux — thus I don’t have to pay for any hosting because the router is anyway always online. My provider gives to my router some public IP which is changes sometimes: maybe about once per week. That is fine for me but I have to change it manually. Of course I can buy a static public IP from my internet provider but my goal is to have cheapest as possible website. So I need to automatically and periodically update the DNS A record with my current IP. To solve this problem people uses Dynamic DNS (DDNS) which is de facto some pseudo protocol when router itself constantly registers it’s current IP on the DNS server. Most routers already have support of some DDNS providers where most popular are DynDNS.com and NO-IP.com or even manufactures like ASUS may have their own DDNS. Gamers and owners of IP cameras very often using this. Unfortunately my DNS registrar doesn’t support DDNS protocol so I have to use some another. A good news is that OpenWrt already have a package ddns-scripts witch supports a lot of servers. I checked almost all DDNS providers that are supported my and choose DuckDNS.org DynDNS.com looks like one of the first DDNS providers and some other even tries to implement it’s API. But it’s paid and that’s not acceptable for me to because with the same money I just can buy a static IP. NO-IP.com have some strange API problems with refreshing IP so there is even a separate script for OpenWrt `ddns-scripts_no-ip_com`. In the same time DuckDNS looks like was made by programmers for programmers. It allows to quickly register with Google account then they give you a generated random token instead of password and they have a good documentation. So the API is so simple that I even was wondered why it was created the ddns-scripts package. In fact, all what you need to do is to login to your OpenWrt LUCI admin panel, then open System / Scheduled Tasks and add the following line: «` * */4 * * * wget -4 -q -O /dev/null http://www.duckdns.org/update/{YOURDOMAIN}/{YOURTOKEN} «` i.e. each 4 hours you will send a HTTP get request to DuckDNS. Then you can check logs of cron task in syslogs: System / System Logs. For example for my domain stokito.duckdns.org: «` Mon Apr 22 18:52:00 2019 cron.info crond[12903]: USER root pid 14005 cmd wget -4 -q -O /dev/null http://www.duckdns.org/update/stokito/6c5se9d3-5220-440-b46-6873f9a «` or you can do the same with command line: «` ssh root@192.168.1.1 root@OpenWrt:~# crontab -e root@OpenWrt:~# /etc/init.d/cron enable root@OpenWrt:~# /etc/init.d/cron restart «` Here note that I enabled the cron service just to be sure. Then your router will be accessible with the new domain. For example for my domain «` $ dig stokito.duckdns.org ; DiG 9.11.5-P1-1ubuntu2-Ubuntu stokito.duckdns.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<>HEADER<<- opcode: QUERY, status: NOERROR, id: 41868 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;baba.stokito.com. IN A ;; ANSWER SECTION: stokito.com. 24 IN CNAME stokito.duckdns.org. stokito.duckdns.org. 24 IN A 77.122.189.118 ;; Query time: 212 msec ;; SERVER: 176.103.130.131#53(176.103.130.131) ;; WHEN: Mon Apr 22 23:55:45 EEST 2019 ;; MSG SIZE rcvd: 129 «` You can see that stokito.com was firstly resolved to CNAME stokito.duckdns.org. which then was resolved to IP 77.122.189.118.

Реклама

Nginx Plus Docker Image + lua-resty-openidc for OAuth termination

I need a Docker image with Nginx Plus and configured lua-resty-openidc to use Keycloak OAuth provider.
I made it based on this article Deploying NGINX and NGINX Plus with Docker but there was few additional non trivial steps so here is my result.
Create a folder with nginx plus repo keys (nginx-repo.crt and nginx-repo.key)
Then create a Dockerfile with the following content:

FROM ubuntu:artful

# Download certificate and key from the customer portal (https://cs.nginx.com)
# and copy to the build context
COPY nginx-repo.crt /etc/ssl/nginx/
COPY nginx-repo.key /etc/ssl/nginx/

# Install NGINX Plus
RUN set -x \
  && apt-get update && apt-get upgrade -y \
  && apt-get install --no-install-recommends --no-install-suggests -y apt-transport-https ca-certificates \
  && apt-get install -y lsb-release wget \
  && wget http://nginx.org/keys/nginx_signing.key && apt-key add nginx_signing.key \
  && wget -q -O /etc/apt/apt.conf.d/90nginx https://cs.nginx.com/static/files/90nginx \
  && printf "deb https://plus-pkgs.nginx.com/ubuntu `lsb_release -cs` nginx-plus\n" | tee /etc/apt/sources.list.d/nginx-plus.list \
  && apt-get update && apt-get install -y nginx-plus nginx-plus-module-lua nginx-plus-module-ndk luarocks libssl1.0-dev git

RUN set -x \
  && apt-get remove --purge --auto-remove -y \
  && rm -rf /var/lib/apt/lists/*

# Forward request logs to Docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
  && ln -sf /dev/stderr /var/log/nginx/error.log

RUN luarocks install lua-resty-openidc
RUN luarocks install lua-cjson
RUN luarocks install lua-resty-string
RUN luarocks install lua-resty-http
RUN luarocks install lua-resty-session
RUN luarocks install lua-resty-jwt

EXPOSE 80

STOPSIGNAL SIGTERM

CMD ["nginx", "-g", "daemon off;"]

Here you can see that we installing not only nginx-plus but also nginx-plus-module-lua and nginx-plus-module-ndk modules which are needed to run lua-resty-openidc.
Since open lua-resty-openidc is distributed via luarocks package manager we need to install it too and then install all needed packages via luarocks. For lua-crypto dependency you need to install libssl1.0-dev package with OpenSSL headers and for some other package we needed git, don't ask me why, I have no idea.
FYI: openidc is installed into file /usr/local/share/lua/5.1/resty/openidc.lua

Then you need to build an image with

docker build --no-cache -t nginxplus .

If you have a Docker Registry inside your company you can publish the image there:

docker tag nginxplus your.docker.registry:5000/nginxplus
docker push your.docker.registry:5000/nginxplus

Not you have an image and you can run it. All you need is to mount you server config into /etc/nginx folder. Consider you have a docker-compose.yml file with the following content:

version: '3'
services:
  gw-nginx:
    image: your.docker.registry:5000/nginxplus
    container_name: gw-nginx
    volumes:
      - ~/gateway/etc/nginx/nginx.conf:/etc/nginx/nginx.conf
      - ~/gateway/etc/nginx/conf.d/:/etc/nginx/conf.d/
      - /etc/localtime:/etc/localtime
    ports:
      - 80:80
  gw-keycloak:
    image: jboss/keycloak
    container_name: gw-keycloak
    volumes:
      - /etc/localtime:/etc/localtime
    ports:
      - 8080:8080
    environment:
      - KEYCLOAK_USER=root
      - KEYCLOAK_PASSWORD=changeMePlease
      - PROXY_ADDRESS_FORWARDING=true

Now create a gateway folder:
mkdir ~/gateway
cd ~/gateway
And plase nginx.conf file into ~/gateway/etc/nginx/nginx.conf. The most important is to place this two lines:

...
http {
  resolver yourdnsip;
  lua_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
  lua_ssl_verify_depth 5;
  ...
}

For some reason without this lines Lua scripts can’t resolve upstreams.

Create ~/gateway/etc/nginx/conf.d/default.conf file and configure it as descibed in restly-openidc documentation.
Finally you can run it with docker-compose up -d command.

Letsencrypt

How to: LetsEncrypt HTTPS on OpenWRT with uhttpd

My old router TP Link WRN740N hosting my homepage stokito.name and it’s too small to handle full LetsEncrypt certbot installer and OpenSSL. So if you want to enable HTTPS you have to run certbot on some other machine and then upload to router.
Here I would like to show how I did that.

Manual installation

The fisrt step is to use manual certs installation from my laptop and renew them after 3 month. Actually this can be automated too latter.

Now, lets generate certs for your domain:

$ sudo certbot certonly --manual --preferred-challenges http

Answer all the questions and it will ask you to upload a file to your router:

-------------------------------------------------------------------------------
Create a file containing just this data:

1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KuC8g.OKKBaAC2SgfXHQyvgKrLkn3zyCNH82xHgKsMg9OQQJE

And make it available on your web server at this URL:

http://www.stokito.name/.well-known/acme-challenge/1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KuC8g

-------------------------------------------------------------------------------
Press Enter to Continue

You need to create the folder on router:

# mkdir -p ./.well-known/acme-challenge

Then upload the files via SCP from your computer to router:

$ echo "1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g.OKKBaAC2SgfXHQyvgKrLkn3zyCNH82xHgKsMg9OQQJE" > 1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g

$ scp ./1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g root@192.168.1.1:/www/.well-known/acme-challenge/1Fyw2Q3IARaG0G6RVUJS587HG_Ou6pKpBLZC-_KeC4g

BTW, there is no any analogue of WinSCP for Linux but you can try to run it on Wine.

Then go back to certbot and press Enter. It will check that files are in place and accessible from web.

 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/stokito.name/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/stokito.name/privkey.pem
   Your cert will expire on 2018-01-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again.

It generated two files: privkey.pem and fullchain.pem (which is not public key!)

Finally, you need to convert the private key and the certificate from the ASCII-armored PEM format to the more economical binary DER format used by uhttpd:

openssl rsa -in privkey.pem -outform DER -out uhttpd.key
openssl x509 -in fullchain.pem -outform DER -out uhttpd.crt

Upload them to the router

$ scp uhttpd.crt root@192.168.1.1:/etc/uhttpd.crt
$ scp uhttpd.key root@192.168.1.1:/etc/uhttpd.key

On the router you need to install uhttpd-mod-tls:

# opkg update
# opkg install uhttpd-mod-tls

edit /etc/config/uhttpd as described in docs i.e. like this

config uhttpd 'stokito.name'
    list listen_http '31.172.137.103:80'
    list listen_https '31.172.137.103:443'
    option redirect_https '1'
    option home '/www'
    option rfc1918_filter '0'
    option cert '/etc/uhttpd.crt'
    option key '/etc/uhttpd.key'

Here 31.172.137.103 is my public static IP.
Note that 443 port should be opened in /etc/config/firewall:

config rule
    option target 'ACCEPT'
    option src 'wan'
    option proto 'tcp'
    option dest_port '443'
    option name 'HTTPS'

Then restart the firewall and uhttpd server:

# /etc/init.d/firewall restart
# /etc/init.d/uhttpd restart

Now try your site in browser. But please check your site latter: I noticed that uhttpd was down but after restart it worked well.

Renewing cert

… So it passed 3 months and my cert got expired and I need to renew it. It’s funny that today is an Old New Year and from window I hear some concert on my street.
This time I decided not to use manual mode and use standalone mode instead: certbot starts itself an https server on 443 port and I need to shut down webserver on my router and enable 443 port forwarding from router to my laptop.

So let’s do that:
1. Connect to router and stop uhttpd service:

$ ssh root@192.168.1.1
# /etc/init.d/uhttpd stop
  1. Enable 443 port forwarding. Download firewall config from router:
$ scp root@192.168.1.1:/etc/config/firewall ./
  1. Edit and comment out current rule for 443 port. If you used that one that I mentioned before then:
# temporarry comment out the rule
#config rule
#  option target 'ACCEPT'
#  option src 'wan'
#  option proto 'tcp'
#  option dest_port '443'
#  option name 'HTTPS'

Then add a HTTPS forwarding rule:

config 'redirect'
    option 'name' 'HTTPS_to_laptop'
    option 'src' 'wan'
    option 'proto' 'tcp'
    option 'src_dport' '443'
    option 'dest_ip' '192.168.1.144'
    option 'dest_port' '443'
    option 'target' 'DNAT'
    option 'dest' 'lan'

Where 192.168.1.144 is the ip of the your laptop. Run ifconfig to see it.

  1. Upload the new firewall config to router:
$ scp ./firewall root@192.168.1.1:/etc/config/
  1. Now restart firewall service on router:
# /etc/init.d/firewall restart
  1. Now your laptop’s 443 port is exposed to the world. So lets tun certbot:
$ sudo certbot certonly --standalone --preferred-challenges tls-sni --cert-name=stokito.name
  1. Then convert the keys do DER format and upload to router as was described above. Then disable forwardind firewall rule and rollback previous and restart firewall and uhttpd.

Now you certs was renewed.

Transliteration to ASCII

If you need to make a translitartion from any language to ASCII symbols you can use a Transliterator from ICU4J.

private static final String TRANSLITERATION_RULE = "Any-Latin; Latin-ASCII";

private static String transliterate(String name) {
    String ascii = TRANSLITERATOR.transliterate(name);
    // Some Russian names may contain Soft Sign ( Ь ) and ( Ъ ) that may cause error http://sourceforge.net/p/icu/mailman/message/34413588/
    ascii = ascii.replaceAll("[ʹʺ]", "");
    return ascii;
}

ICU Transform Demonstration
1) Select «Names» from «Inset sample» combo box.
2) Insert the rule «Any-Latin; Latin-ASCII» to the «Compound 1» fields.
3) Press «Transform» button

Also a good example:
How do I convert Chinese characters to their Latin equivalents?

What are the system Transliterators available with ICU4J?

[HOWTO] WYSIWYG with Grails + CKEditor + Spring Security

Add last version of Grails CKEditor plugin to BuildConfig.groovy

grails.project.dependency.resolution = {
...
    plugins {
        ...
        compile ":ckeditor:4.4.1.0"
    }
}

Plugin version corresponding to bundled CKEditor version, and may be outdated.
To update it, and you use assets pipeline plugin, download CKEditor and unpack to grails-app/assets/ckeditor.
Then you should add dependency to application.js like this

//= require jquery-1.11.1.min
//= require jquery-ui.min
//= require_tree plugins
//= require_tree globalize
//= require bootstrap
//= require ../ckeditor/ckeditor
//= require_self

If CKEditor version is good enough for you, just put ckeditor:resources tag into head of view:

<html>
<head>
    <title>Example</title>
    <ckeditor:resources/>
</head>

Then you can insert CKEditor in form with tag ckeditor:editor. For example editor of article content:

<ckeditor:editor name="content" height="400px" width="80%" userSpace="${currentUser.id}">${article.content}</ckeditor:editor>

Here used attribute userSpace with value of current user's id. It makes all user images upload to server separated to their own folders.

But you should also restrict access to users spaces. If you use Spring Security (S2) plugin you can create custom filter with command grails create-filters OfmSecurity:

class OfmSecurityFilters {
 def springSecurityService
 def filters = {
 all(uri: '/ck/ofm/**') {
 before = {
 if (springSecurityService.currentUser?.id != params.space?.toLong()) {
 redirect(controller: "user", action: "login")
 return false
 }

 }
 after = { Map model ->

 }
 afterView = { Exception e ->

 }
 }
 }
}

[Чтиво] Три очень классные практические статьи-инструкции для программистов

Периодически почитываю что накопилось в закладках. Вот отличные прагматичные статьи вкурив которые сразу получите +2 к экспе девелопера.
Экстремальное программирование: Pair Programming чёткая инструкция по делу что такое парное программирование как его делать на практике и самое главное как его НЕ делать.

Парное программирование в аутсорсинге: достижение взаимопонимания с техническими специалистами заказчика Классная статья которую обязан прочесть каждый программист-аутсорсер. Вывод: если у вас есть часть удалённой команды то парно педалить код вы просто обязаны.

Практика рефакторинга в больших проектах в сотый раз о вечном. Для тех кто ещё халтурит и не прочёл замечательную книгу Working Effectively with Legacy Code

[Grails] Mock MySQL database in test environment

Grails supports different data sources configured per environment.
If you use MySQL database in production and want to write integration tests you may prefer H2 database because it simpler.
For example it can automatically create in-memory database from the domain classes of GORM model.
And after running tests it will drop this database when application will shutting down.
H2 allows you to use full set of SQL commands, but they may differ from MySQL.
To imitate MySQL behavior you need to add options Mode=MySQL;DATABASE_TO_UPPER=FALSE;IGNORECASE=TRUE to connection string.

From documentation:

MySQL Compatibility Mode

To use the MySQL mode, use the database URL jdbc:h2:~/test;MODE=MySQL or the SQL statement SET MODE MySQL.

  • When inserting data, if a column is defined to be NOT NULL and NULL is inserted, then a 0 (or empty string, or the current timestamp for timestamp columns) value is used. Usually, this operation is not allowed and an exception is thrown.
  • Creating indexes in the CREATE TABLE statement is allowed using INDEX(..) or KEY(..). Example: create table test(id int primary key, name varchar(255), key idx_name(name));
  • Meta data calls return identifiers in lower case.
  • When converting a floating point number to an integer, the fractional digits are not truncated, but the value is rounded.
  • Concatenating NULL with another value results in the other value.

Text comparison in MySQL is case insensitive by default, while in H2 it is case sensitive (as in most other databases). H2 does support case insensitive text comparison, but it needs to be set separately, using SET IGNORECASE TRUE. This affects comparison using =, LIKE, REGEXP.

But raw SQL may not work, because MySQL field names are case sensitive, GORM expect them lowercased, but by default H2 creates them uppercased.
To fix that you need to set property DATABASE_TO_UPPER to FALSE

Database setting DATABASE_TO_UPPER (default: true).
Database short names are converted to uppercase for the DATABASE() function, and in the CATALOG column of all database meta data methods. Setting this to «false» is experimental. When set to false, all identifier names (table names, column names) are case sensitive (except aggregate, built-in functions, data types, and keywords).

So final Config.groovy will looks like:

environments {
    test {
        dataSource {
            dbCreate = 'create'
            url = 'jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;MODE=MySQL;DATABASE_TO_UPPER=FALSE;IGNORECASE=TRUE'
            pooled = true
            driverClassName = org.h2.Driver.class.canonicalName
            dialect = H2Dialect
            username = 'sa'
            password = ''
            logSql = true
            formatSql = true
            pooled = true
        }
    }
}