Creating a DIY 433 MHz ESP8266-based Home Automation bridge to switch DIP remote control outlets

A couple of years ago I built a pretty basic smart home application allowing me to control my remote controlled sockets via an Android app or a Web Extension. It’s based on the rcswitch library run on an Apache. The 433 MHz signals are sent by a FS1000A transmitter hooked up via GPIO to a Raspberry Pi. The whole setup lied on the ground in a corner of my apartment behind a curtain next to my network wall jack. Now where we have just moved to a nice new and twice as big home, I needed a solution which could be placed in the middle of all rooms to allow the rather weak 433 MHz signals to reach every receiver. Additionally, I wanted to get rid of having to maintain a full Ubuntu server, which only serves as a light switch for the most part.

Firefox WebExtension used to turn on desk lamp
Firefox WebExtension used to turn on desk lamp

Around that time, I stumbled upon the ESP8266: a low-cost WiFi microchip with full TCP/IP stack and microcontroller capability – ideally soldered on a NodeMCU or Wemos D1 for the maximum level of convenience. Arduino and Wifi: a whole new world of IoT-possibilities. Once you’ve added the board manager to your Arduino IDE, you can use those tiny boards just like an ordinary Arduino. As the Arduino WebServer library can turn a NodeMCU development board into a light-weight HTTP server and the rcswitch library is also available on Arduino, I decided to put both – NodeMCU and FS1000A – into a junction box to create a DIY 433 MHz RF WiFi bridge.

To reach the bridge you should either assign a static IP or use mDNS. Keep in mind that mDNS is not supported by all operating systems out of the box. If in doubt, use a static IP.  To make the bridge accessible from outside your home network, you need to open and forward a port on your router (port 80 by default and can be changed in line 10). It’s recommend to secure any connection made through the public internet. By the time I was writing the script, there hasn’t been a HTTPS server implementation around. However, I found HelloServerBearSSL while writing this article. It looks very promising and is definitely worth a try.

433Mhz RF WiFi Bridge Junction Box opened and closed
433 MHz RF WiFi Bridge Junction Box opened and closed

This project works with simple DIP-switch remote outlets only. It became quite hard to get the “old” DIP outlets as most producers switched to the “new” outlets, which use a button on the receivers to “learn” a signal. There is a NewRemoteSwitch library to deal with them. But as I just recently found one last triple pack in a dollar store, I am stocked until I will eventually move to Wifi controlled outlets.

Enough talk. A typical request contains the five digit system code, five digit unit code and a binary power code separated by comma. You can also concatenate multiple commands using semicolon. A sample request to switch on outlet A and switch off outlet B would look like this:

x.x.x.x/switch?command=10010,00001,1;10010,0010,0

Here is the code. Further down is a download link.

DOWNLOAD: 433MHzWifiBridge Project

 

Generating random alphanumeric profanity free codes using pthreads in PHP

A friend of mine recently forwarded me an offer he received for the generation of 50 million random codes: alphanumeric, 10 characters long, unique and not containing any profane words. Price tag: same as a brand new mid-range car. Lol. Hold my beer, I’ll do this. 🙂

Not too long after writing the first line, the SQLite table filled with codes in a breeze…. until I cranked up the profanity check from a couple of test badwords to a real world scenario list of around 2500 finest German swear words. It absolutely killed the performance, while the CPU utilization did not even hit the 30% mark. Being to lazy to rewrite everything in e. g. Java, I started to take a look at ways to bring multi-threading to PHP. This led me to pthreads, a project providing multi-threading based on Posix Threads. Motivation follows action, action follows laziness and voilà: the code generator is now able to utilize all available processing power. Combined with a few tweaks of the bad word dictionary, it dramatically reduced the time needed to finish the job. A test run on my old i7 4something took two and a half hours (using this English profanity list and requiring a minimum Shannon entropy of theoretically 2.2 bits per character).

The whole project and its output can be downloaded below. Make sure to install pthreads first. The script configuration is done in the Config.php. Also note that pthreads projects can be run via CLI only.

A couple of learnings made:

* Use a multi-threading language in the first place when thinking about solving highly repetitive tasks.
* Use random_int() instead of rand(). Using rand() will quickly lead you to duplicate codes as it does not generate cryptographically secure values.
* Create objects, that need to be passed into a pthreads worker, in the calling context and keep a reference. Objects created in a thread scope constructor will be destroyed to avoid memory issues.
* Combining multiple SQL INSERTs to one transaction will take way less time than inserting one by one.
* Having an idea about the statistical probability of hitting a duplicate code or unwanted word, helps balancing out the efforts taken to avoid them. Keep in mind that every constraint will make it easier to guess a code.

DOWNLOAD: CodeGenerator Project
DOWNLOAD: 50 mio codes (1.1 GB, zipped)

Optimizing the Amazon rating histogram table

Amazon recently updated their website to show only the percentage of ratings behind each rating bar. While this usually comes in quite handy, it is counterproductive when there is only a small number of ratings. So I wrote a Greasemonkey script, which brings back the absolute number of ratings behind each bar and moves the percentage directly onto it. While working on this, I noticed that there is a very tiny little tooltip triangle right behind the average rating. It says that the shown average rating is not the arithmetic one, but instead a score that has been adjusted based on certain parameters (e. g. age, helpfulness, verified purchases) by Amazon. The first thought is obvious: “Sneaky Amazon. Nice trick to increase the ratings and drive sales.”. Turns out the shown average ratings are often lower than the arithmetic average. At least for the couple of samples I took. Could also be an approach to control sales by systematically devaluing certain products. Who knows… At least interesting enough to put it as a follow up project on the “maybe next winter” list. Still, there remains a bland taste and a strong feeling that Amazon ratings keep on becoming less and less trustworthy.

Note: Amazon keeps changing its markup quite often. The script might already have stopped working at the time you are reading this.

DOWNLOAD

 

Redirect multi-page news articles to a single page view using Greasemonkey

A lot of news websites split their articles into multiple pages. In theory this drives page impressions and thus ad revenue. In practice it is just annoying. The Greasemonkey script below automagically redirects you to a news article’s full page (in this case on zeit.de). It uses a MutiationObserver to wait for the pager UI element. The script kicks in once it appears – no need to wait until their page (or ads?!) have been fully loaded. 🙂

You can easily modify it:

1. Include all pages that might contain a multi-page article
2. Exclude all pages that might lead to a loop (especially the target site) or that will not contain a pager element (MutiationObserver can easily add a couple of milliseconds of loading time on complex pages)
3. Change the class name of the pager element to the one used by your site
4. Change the target location to the one you want to be redirect to

 

Ready-to-install scripts for:

golem.de
heise.de
zeit.de

The easiest way to send basic HTTP POST or GET requests using PHP

The easiest way to send basic HTTP POST or GET requests is using PHP’s built in file_get_contents() function in conjunction with HTTP context options:

Further reading:

file_get_contents
HTTP context options

Using a ML8511 UV sensor and an Arduino Nano to test UV-filtering properties of sunglasses

Do cheap freebie sunglasses really block UV light or are they mostly just toys, which pose a serious health risk to your eyes? That’s the question I asked myself when coming across one of my wife’s cheesy glasses. I always warned her, but never could proof the potential risk of increased UV exposure caused by non-blocking tinted glasses. Until now…

After buying one of those cheap Arduino Nano clones (oh the irony), I started experimenting with all kind of sensors. One of them was the ML8511. This sensor detects 280 – 390 nm light most effectively. This wavelength is categorized as part of the UVB spectrum and most of the UVA spectrum. Overexposure to UVA radiation has been linked to the development of certain types of cataracts, and research suggests UVA rays may play a role in development of macular degeneration.

The setup was straight forward: I hung an UV LED torch over the sensor using my helping hand. The emitted 395 nm light is slightly out of range, but the torch has proven to be a reliable source of detectable UV light. Pointing the beam directly at the photo resistor was crucial as the amount of measurable UV light decreases rapidly on the beam’s edges. I used a sketch from Sparkfun to read the sensor’s output. The unit measure wasn’t really necessary as I was mainly interested in the relative amount of absorbed UV light. But having the Milliwatts per square Centimeter came in handy. The problem was that the script outputted -1 mW/cm² when being in an UV-free environment. The solution was unexpected: The voltage of the “supposed to be 1% accurate” 3v3 Nano output was in fact only 10% accurate and came out as being 3.61 volts. Seems like testing cheap sunglasses using even cheaper tools isn’t the best idea. However, the sketch’s output can be calibrated by adjusting the hard-coded reference variable in the script to the actual reference voltage.

All tested sunglasses absorbed a fair amount of UV light. The best pairs filtered the UV light nearly completely (meaning below a level that can be detected by an ML8511 in this particular setup and ignoring the minor UV halo around the frame due to the sensor not being fully covered by the glass). The rest ranged between letting 1 – 10% of UV irradiation to pass through – proving that all sunglasses had UV-filtering properties. Given that a good UV protection can be bought for less than 10 Euros (the best pair tested), it is probably a good idea to not use freebie sunglasses if you have a bad gut feeling. As general advice, make sure that you buy your glasses from trusted retailers (optician, pharmacy, supermarket, etc.) or let them be tested.

Seeing is believing. 😉

 

UV test setup with ML8511 and Arduino Nano R3
Serial monitor output of the sketch used

 

 

Reference and further reading:

https://learn.sparkfun.com/tutorials/ml8511-uv-sensor-hookup-guide
https://en.wikipedia.org/wiki/Ultraviolet
http://www.canadianjournalofophthalmology.ca/article/S0008-4182(17)30495-7/fulltext

The straight forward way to backup all your phone’s files using the Android Debug Bridge

To copy all files from your Android phone’s internal storage to the current local dir using adb:

(‘-p’ to display the transfer progress)
(‘-a’ means copy timestamp and mode)

 

The whole adb help for the sake of completeness:

 

 

Stepscout: A tool to search for jobs and apartments at the same time

I think living near one’s workplace is a great benefit for a good work-live balance. If you are living in your current city for a while, you probably already know where the best places to live and work are. But if you are starting to search for one or the other, you might ask yourself which neighborhood is the best to reduce the daily commute to a minimum. To find answers to this question I build Stepscout. It leverages the Stepstone and ImmobilienScout24 API’s to find jobs & apartments in one go and marks them together on a Google Map. This visualization helps to find job clusters, apartment clusters or ideally job-apartment-clusters.

The trick in this project was to make use of Google’s Places API to search for latitude and longitude of every company in the Stepstone result response. Even though Stepstone’s response JSON contains fields for geographic coordinates, they are empty or filled with generic values for the most part.

If you are living in Germany, you can check out the tool here or get a first impression below.

Boilerplate for a basic PHP cURL POST or GET request with parameters on Apache

cURL is a library for transferring data using various protocols – in this case most importantly HTTP POST and GET. PHP installed on a Linux distribution or as part of XAMPP uses libcurl. If you haven’t enabled cURL yet, open your php.ini and remove the semicolon at the beginning of this line:

You will find the location of your php.ini in the output’s first line when running

on the command line or by using the XAMPP control panel on a Windows machine. Click the ‘Config’ button next to the Apache module and select ‘PHP (php.ini)’ from the context menu. Save the changes and restart Apache – either by pressing ‘Stop’ & ‘Start’ on the XAMPP control panel or by using the Linux command line:

If cURL for PHP isn’t installed, run

 prior to the step above.

You’ll find further information on how to use cURL here: http://php.net/manual/en/book.curl.php

This boilerplate wraps cURL in a simple function with four parameters: request type, url, parameters and headers. The first snippet contains comments for every step. The second snippet is exactly the same code but without any comments.

Commented boilerplate:

 

And the raw template without comments:

 

 

Disable autoplay on Youtube’s new 2017 material design release

I happened to receive Youtube’s new 2017 desktop material design when I was watching videos the other day. All in all a great redesign which closes the gap to their other channels. You can force the new design by running the command below in your Firefox’s console’s command line (press [CMD] + [SHIFT] + [K]). Reload the page when done.

Unfortunately, the Firefox Greasemonkey script I used to disable Youtube’s autoplay feature does not work with their new site. Just removing the autoplay toggle’s node from the DOM does not do the trick anymore. My investigations brought to light that the f5 property of the PREF cookie is used to toggle the autoplay feature under the hood. f5=30000 is the default value to disable autoplay and f5=20000 the default to enable it. So I built a new Greasemonkey script which reads the existing PREF cookie, looks for the f5-property and sets it accordingly (or adds it if not present). At the same time all existing values are preserved. Additionally, the autonav_disable cookie is set. It was the first thing I found during my investigations which made me think “easy….”. Anyways, it turned out this cookie is not used to control autoplay. Not sure what it does, but I set it just to play safe. Finally a MutationObserver is used to wait for the autoplay toggle and remove it once it is loaded. DOMContentLoaded did not help as it seems like the node is added afterwards. To install the script, first get Greasemonkey for Firefox here . Once Greasemonkey is installed, click here to install the Userscript or paste the code below into your own script. Force reload Youtube by pressing [CMD] + [F5] after successfully installing the script.

Bonus: To hide Youtube’s cookie consent bar, uncomment the first occurrence of setCookieConsentHideCookie(); in the script.