Digital Ocean private LAN is totally useless

Digital Ocean is offering a private LAN for internal communication between the VMs, and they claim it’s isolated from other customers. You get some random addresses within 10.133.0.0/16 (or maybe some other range), and they can talk to each other on dedicated virtual NICs.

But that’s it. You cannot run OSPF because multicast packets are not let through. Even if you manage configuring direct neighbors in OSPF, it renders useless because the private LAN does not allow packets with destination IP addresses outside of the LAN range. So, any kind of routing with next hop in the private LAN would not work.

Too bad guys, very disappointed. So, we need to resort to Tinc VPN for internal routing, and this private LAN doesn’t make any sense.

WAN backup routing via LTE

A Linux device, such as PC Engines APU, can be equipped with an LTE modem, but sometimes it’s desirable to use the mobile connection only if the wired connection is unavailable.

The following scenario is for Debian 9 on an APU box, but it’s also applicable to any other Linux device.

The DHCP client is tweaked to ignore the DNS server addresses that are coming with  DCHP offer. Otherwise, the LTE provider may provide DNS addresses that are not usable via the ethernet WAN link.

The “ifmetric” package allows setting metrics in interface definitions in Debian. This way we can have two default routes with a preferred metric over LAN interface. The default route with lower metric is chosen for outbound traffic.

The watchdog process checks availability of a well-known public IP address over each of the uplinks, and shuts down and brings up again the corresponding interface. It only protects from next-hop failures. If you want to protect from failures in the whole WAN service, you need to increase the Ethernet port metric if it fails, and then start checking the connectivity, and lower the metric when it’s stable again.

Also the second NIC on the box is configured to provide DHCP address and to NAT all outbound traffic.

Detailed installation instructions are presented here: https://gist.github.com/ssinyagin/1afad07f8c2f58d9d5cc58b2ddbba0a7

 

Ubiquiti EdgeRouter X, a powerful $50 device

Ubiquiti EdgeRouter X is a tiny and cheap (around $50) router with a decent amount of memory: 256MB RAM and 256MB flash. The router offers 5 GigE copper ports, and there’s also a model with an additional SFP port. The device is produced since 2014, and it’s still up to date and a good value for money.

On hardware level, the device consists of a Gigabit Ethernet switch, with one GigE port attached to the MIPS CPU and used as a 802.1q trunk. Also inside the enclosure, serial console port is available for easy debugging or manipulating the boot loader.

The router comes with stock Ubuquiti software which is based on Debian Wheezy, so many files are from 2013-2014. OpenVPN package is pre-installed, but only version 2.3 is available. The software offers a nice GUI and SSH access.

OpenWRT provides excellent support for this hardware. The router is able to perform IP routing at more than 400Mbps (I haven’t tested it with back-to-back connection, so I don’t know the limit).

Also with OpenVPN 2.4 that is available in up-to-date OpenWRT packages, the box performs at up to 20Mbps with 256-bit AES encryption, and at about 55Mbps with encryption and authentication disabled.

In default OpenWRT configuration, the switch port 0 is dedicated to WAN link, and ports 1-4 are used as a LAN bridge. The WAN link acts as a DHCP client, and LAN is configured with DHCP service in 192.168.1.0/24 range. The command-line configuration utilities are quite straightforward, and there’s a Web UI as well.

OpenVPN scenarios and scripts

Here’s a new repository for OpenVPN deployment scenarios and example configurations:

https://github.com/txlab/ovpn-scripts

At the moment it lists two scenarios with configuration generation scripts:

1. routed VPN for remote management
2. bridged VPN for anti-censorship isolation of a home LAN

 

SIMCom SIM7100E LTE modem

SIMCom SIM7100E is a recent LTE modem released by Simcom. It’s approximately $20 cheaper than Huawei LTE modem, and also it provides USB voice function, so it could be integrated with FreeSWITCH mod_gsmopen module (this needs development).

My set of udev rules and chat scripts is updated with SIM7100E information, and here’s a copy:

cat >/etc/udev/rules.d/99-wwan.rules <<'EOT'
# SIMCom SIM7100
SUBSYSTEM=="tty", ATTRS{idVendor}=="1e0e", ATTRS{idProduct}=="9001", SYMLINK+="ttyWWAN%E{ID_USB_INTERFACE_NUM}"
SUBSYSTEM=="net", ATTRS{idVendor}=="1e0e", ATTRS{idProduct}=="9001", NAME="lte0" 
EOT

cat >/etc/chatscripts/sunrise.SIM7100 <<'EOT'
ABORT BUSY
ABORT 'NO CARRIER'
ABORT ERROR
TIMEOUT 10
'' 'AT+CFUN=1'
OK 'AT+CMEE=0'
OK 'AT+CGDCONT=1,"IP","internet"'
OK '\dAT\$QCRMCALL=1,1'
OK
EOT

cat >/etc/chatscripts/gsm_off.SIM7100 <<'EOT'
ABORT ERROR
TIMEOUT 5
'' 'AT\$QCRMCALL=0,1'
OK AT+CFUN=0
OK
EOT

cat >/etc/network/interfaces.d/lte0 <<'EOT'
allow-hotplug lte0
iface lte0 inet dhcp
 pre-up /usr/sbin/chat -v -f /etc/chatscripts/sunrise.SIM7100 >/dev/ttyWWAN02 </dev/ttyWWAN02
 post-down /usr/sbin/chat -v -f /etc/chatscripts/gsm_off.SIM7100 >/dev/ttyWWAN02 </dev/ttyWWAN02
EOT

 

FriendlyELEC NanoPi NEO Plus2

NanoPi NEO Plus2 is a brand new board released by FriendlyELEC. It’s slightly bigger than the NEO2 board, and packed with much more cool stuff: 1GB RAM, Wifi+Bluetooth module, and 8GB eMMC chip. It has also two USB2.0 port connected to independent USB controllers.

The NanoPi NEO Plus2 Basic Kit accompanies the board with an acrylic enclosure, and the first orders are delivered with an UART USB adapter. They also listed an antenna, but I did not receive it in my kit. Anyway I have a better option, a flat self-adhesive antenna like this one.

The acrylic enclosure is about two times thicker than that for NEO boards, and it also has a hole for antenna mount. I added 8 pieces of M2.5 washers and 4 M3 pillars to the original design, to make it more long-lasting. The photo below has the UART adapter plugged in.

Armbian still needs some work to be done to support this new board. But the Ubuntu image that is available from FriendlyELEC is quite enough to demonstrate all the hardware capabilities. Unlike Armbian, it does not mount /tmp and /var/log as tmpfs, so the SD card may experience a faster wearing.

Installing vSphere Replication from Linux CLI

tested with VCSA 6.1 and vSphere replication 6.1.1. OVF Tool 4.2.0 is installed on a Debian Jessy machine.

ovftool --acceptAllEulas -ds=datastore1 \
-n=<VMname> --ipAllocationPolicy=fixedPolicy \
--prop:password='*******' \
--prop:ntpserver=******* \
--vService:installation=com.vmware.vim.vsm:extension_vservice \
vSphere_Replication_OVF10.ovf  vi://vcenter01.domain.com/DC1/host/Cluster1/

 

Acrylic enclosure for FriendlyElec NanoPi NEO2

The NanoPi NEO2 board by FriendlyElec has several options for an enclosure in their webshop. The 3D-printed plastic enclosure is of too poor quality, and it doesn’t fixate the heatsink properly on the CPU.

The acrylic case does not include washers, which makes the whole construct too fragile, as the screws can easily damage the plastic.  Also the M2.5 screws for fixing the heatsink are too short.

So, I added the following components to the design:

• M3*16mm  screws and M3 nuts (4 pieces each)
• M3 washers (24 pieces)

Also the following parts came with the acrylic case:

• M3*6mm screws (4 pieces)
• 6.3mm plastic spacers (4 pieces)
• 25mm female-female M3 spacers (4 pieces)
• 6mm male-female M3 spacers (4 pieces)

As a result, we get a sturdy case that is able to sustain some rough handling, like carrying it in a toolbox among other hardware.

(scratches on my phone camera made the pictures a bit too soft)

Two LTE modems with PC Engines APU3

PC Engines GmbH has recently released a new board, APU3. The difference from APU2 is that two mPCIe slots are suitable for 3G or LTE modems, whereas APU2 had only one such slot. This article explains how to utilize two HUAWEI ME909 LTE modems, and it’s applicable to other modems too.

One of the LTE modems has to occupy the slot which is otherwise usable for mSATA storage. So, the board has to use the SD card for booting, and Voyage Linux is designed for such setup. The scripts in this article are tested against Voyage Linux version: 0.11.0 (Build Date 20170122).

As with APU2, the Linux kernel assigns ttyUSB port numbers randomly, so two ME909 modems produce 10 ttyUSB devices with random numbers which change after a reboot.

The modems have identical serial numbers “0123456789ABCDEF”, and the only thing that allows distinguishing them reliably is the PCI slot number of the corresponding USB controller.

Luckily, APU3 board slots designed for LTE modems, J14 (mSATA/mPCIe 3), and J15 (mPCIE 2), are attached to different USB controllers. The third slot, J16 (mPCIE 1), shares the same USB controller with J15.

USB EHCI Controller at PCI device 00:12.0 is attached to J14, and the controller at 00:13.0 is attached to J15 and J16.

So, the udev rules require a small Shell script that translates DEVPATH variable into the PCI slot and function number, and the resulting string will persistently distinguish the devices attached to USB interfaces in J14 and J15:

cat >/etc/udev/devpath_to_pcislot <<'EOT'   
#!/bin/sh echo ${DEVPATH} | sed -r \
    -e 's,^\/[^\/]+\/[^\/]+\/[0-9af]{4}:[0-9af]{2}:,,' \
    -e 's,\/.+,,' -e 's,\.,,g' 
EOT 

cat >/etc/udev/rules.d/99-wwan.rules <<'EOT'
SUBSYSTEM=="tty", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="15c1", PROGRAM="/etc/udev/devpath_to_pcislot" SYMLINK+="ttyWWAN%c{1}_%E{ID_USB_INTERFACE_NUM}"
SUBSYSTEM=="net", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="15c1", PROGRAM="/etc/udev/devpath_to_pcislot" NAME="lte%c{1}"
EOT

After rebooting, you can see “lte120” and “lte130” network interfaces, and devices suitable for configuring modems: “/dev/ttyWWAN120_02” and “/dev/ttyWWAN130_02”. There are few other TTY interfaces for various purposes, as explained in HUAWEI documentation.

cpio: cap_set_file error when installing httpd RPM inside an LXC container

My physical machine runs Debian Jessie, and it has several LXC containers (mostly Debian and Ubuntu). Now I needed to test some software under CentOS, and I bumped into the following error when installing Apache HTTP server:

Downloading packages:
httpd-2.4.6-45.el7.centos.4.x86_64.rpm                                                                        | 2.7 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : httpd-2.4.6-45.el7.centos.4.x86_64                                                                                1/1 
Error unpacking rpm package httpd-2.4.6-45.el7.centos.4.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec;590112cd: cpio: cap_set_file
  Verifying  : httpd-2.4.6-45.el7.centos.4.x86_64                                                                                1/1 

Failed:
  httpd.x86_64 0:2.4.6-45.el7.centos.4

The thing is, that by default “/usr/share/lxc/config/centos.common.conf” defines the following capability drops:

lxc.cap.drop = mac_admin mac_override setfcap setpcap
lxc.cap.drop = sys_module sys_nice sys_pacct
lxc.cap.drop = sys_rawio sys_time

So, setfcap capability is required in order to install Apache. Use the following lines in your “/var/lib/lxc/NAME/config” to drop previously defined drops and set up a new list:

# flush all defined drops and define a new list
lxc.cap.drop =
lxc.cap.drop = mac_admin mac_override setpcap
lxc.cap.drop = sys_module sys_nice sys_pacct
lxc.cap.drop = sys_rawio sys_time

then restart the container, and “yum install httpd” should run as expected.