Generating random alphanumeric profanity free codes using pthreads in PHP

A friend of mine recently forwarded me an offer he received for the generation of 50 million random codes: alphanumeric, 10 characters long, unique and not containing any profane words. Price tag: same as a brand new mid-range car. Lol. Hold my beer, I’ll do this. 🙂

Not too long after writing the first line, the SQLite table filled with codes in a breeze…. until I cranked up the profanity check from a couple of test badwords to a real world scenario list of around 2500 finest German swear words. It absolutely killed the performance, while the CPU utilization did not even hit the 30% mark. Being to lazy to rewrite everything in e. g. Java, I started to take a look at ways to bring multi-threading to PHP. This led me to pthreads, a project providing multi-threading based on Posix Threads. Motivation follows action, action follows laziness and voilà: the code generator is now able to utilize all available processing power. Combined with a few tweaks of the bad word dictionary, it dramatically reduced the time needed to finish the job. A test run on my old i7 4something took two and a half hours (using this English profanity list and requiring a minimum Shannon entropy of theoretically 2.2 bits per character).

The whole project and its output can be downloaded below. Make sure to install pthreads first. The script configuration is done in the Config.php. Also note that pthreads projects can be run via CLI only.

A couple of learnings made:

* Use a multi-threading language in the first place when thinking about solving highly repetitive tasks.
* Use random_int() instead of rand(). Using rand() will quickly lead you to duplicate codes as it does not generate cryptographically secure values.
* Create objects, that need to be passed into a pthreads worker, in the calling context and keep a reference. Objects created in a thread scope constructor will be destroyed to avoid memory issues.
* Combining multiple SQL INSERTs to one transaction will take way less time than inserting one by one.
* Having an idea about the statistical probability of hitting a duplicate code or unwanted word, helps balancing out the efforts taken to avoid them. Keep in mind that every constraint will make it easier to guess a code.

DOWNLOAD: CodeGenerator Project
DOWNLOAD: 50 mio codes (1.1 GB, zipped)

The easiest way to send basic HTTP POST or GET requests using PHP

The easiest way to send basic HTTP POST or GET requests is using PHP’s built in file_get_contents() function in conjunction with HTTP context options:

$data = http_build_query(
    array(
        'firstKey' => 'firstValue',
        'secondKey' => 'secondValue',
        'thirdKey' => 'thirdValue'
    )
);

$options = array('http' =>
    array(
        'method'  => 'POST',
        'header'  => 'Content-type: application/x-www-form-urlencoded',
        'content' => $data
    )
);

$context = stream_context_create($options);

$result = file_get_contents('https://httpbin.org/post', false, $context);

Further reading:

file_get_contents
HTTP context options

Stepscout: A tool to search for jobs and apartments at the same time

I think living near one’s workplace is a great benefit for a good work-live balance. If you are living in your current city for a while, you probably already know where the best places to live and work are. But if you are starting to search for one or the other, you might ask yourself which neighborhood is the best to reduce the daily commute to a minimum. To find answers to this question I build Stepscout. It leverages the Stepstone and ImmobilienScout24 API’s to find jobs & apartments in one go and marks them together on a Google Map. This visualization helps to find job clusters, apartment clusters or ideally job-apartment-clusters.

The trick in this project was to make use of Google’s Places API to search for latitude and longitude of every company in the Stepstone result response. Even though Stepstone’s response JSON contains fields for geographic coordinates, they are empty or filled with generic values for the most part.

If you are living in Germany, you can check out the tool here or get a first impression below.

Boilerplate for a basic PHP cURL POST or GET request with parameters on Apache

cURL is a library for transferring data using various protocols – in this case most importantly HTTP POST and GET. PHP installed on a Linux distribution or as part of XAMPP uses libcurl. If you haven’t enabled cURL yet, open your php.ini and remove the semicolon at the beginning of this line:

;extension=php_curl.dll
^--- remove this semicolon

You will find the location of your php.ini in the output’s first line when running

php --ini

on the command line or by using the XAMPP control panel on a Windows machine. Click the ‘Config’ button next to the Apache module and select ‘PHP (php.ini)’ from the context menu. Save the changes and restart Apache – either by pressing ‘Stop’ & ‘Start’ on the XAMPP control panel or by using the Linux command line:

sudo service apache2 restart

If cURL for PHP isn’t installed, run

sudo apt-get install php-curl

 prior to the step above.

You’ll find further information on how to use cURL here: http://php.net/manual/en/book.curl.php

This boilerplate wraps cURL in a simple function with four parameters: request type, url, parameters and headers. The first snippet contains comments for every step. The second snippet is exactly the same code but without any comments.

Commented boilerplate:

<?php
 
$url = 'http://httpbin.org/post';
 
$parameters = array(
      'firstKey' => 'firstValue',
      'secondKey' => 'secondValue',
      'thirdKey' => 'thirdValue'
);
 
$headers = array(
      'X-Custom: 123',
      'X-Api-Key: ABC123'
);
 
// fire the request. Access the response object with $response->success and $response->body
$response = request('POST', $url, $parameters, $headers);

     
/**
 * cURL a resource with parameters
 *
 * @param string $requestType The request type POST or GET  
 * @param string $url The request URL
 * @param array|null $parameters An array of request parameters
 * 
 * @return object Response object with $success (bool) and $body (string) property.
 */ 
function request($requestType, $url, $parameters = null, $headers = null){
 
      // instantiate the response object
      $response = new stdClass();

      // check if cURL is enabled
      if(!function_exists('curl_init')){

            $response->success = false;
            $response->body = 'cURL is not enabled.';

            return $response;
      }
 
      // instantiate a cURL instance and set the handle
      $ch = curl_init();
 
      // build http query if $parameters is not null. Parameters with null as value will be removed from query.
      ($parameters !== null) ? $query = http_build_query($parameters) : $query = '';
 
      // POST:
      if($requestType === 'POST'){
 
            // 1 tells libcurl to do a regular HTTP post and sets a "Content-Type: application/www-form-urlencoded" header by default
            curl_setopt($ch,CURLOPT_POST, 1);                
            // add the query as POST body
            curl_setopt($ch,CURLOPT_POSTFIELDS, $query);     
            
      // GET:
      }elseif ($requestType === 'GET') {
      
            // if not empty, add parameters to URL
            if($query) $url = $url . '?' . $query;                
               
      // ELSE: 
      }else{
 
            $response->success = false;
            $response->body = 'request type GET or POST is missing.';

            return $response;
      }
      
      // set the URL
      curl_setopt($ch, CURLOPT_URL, $url);                  
      // tell cURL to return the response body. A successful request will return true if not set.
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);       
      // disable ssl certificate checks. Dirty, insecure workaround for common error "SSL Error: unable to get local issuer certificate". Fix it the correct way and remove the line!
      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);      
      // more options here: http://php.net/manual/en/function.curl-setopt.php
 
      // add headers if present
      if ($headers !== null) curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

      // execute and store the result
      $result = curl_exec($ch);                             
 
      // check if request was successful. If yes, return result. If not, return error and its code.
      if($result !== false){

            $response->success = true;
            $response->body = $result;

      }else{

            $response->success = false;
            $response->body = curl_error($ch);
            $response->error = curl_errno($ch);
      }                  
      
      // close session and delete handle
      curl_close($ch);                                      

      // return response object
      return $response;                                       
}
 
 
 
?>

 

And the raw template without comments:

<?php
 
$url = 'http://httpbin.org/post';
 
$parameters = array(
      'firstKey' => 'firstValue',
      'secondKey' => 'secondValue',
      'thirdKey' => 'thirdValue'
);
 
$headers = array(
      'X-Custom: 123',
      'X-Api-Key: ABC123'
);
 

$response = request('POST', $url, $parameters, $headers);


function request($requestType, $url, $parameters = null, $headers = null){
 
      $response = new stdClass();

      if(!function_exists('curl_init')){

            $response->success = false;
            $response->body = 'cURL is not enabled.';

            return $response;
      }
 
      $ch = curl_init();
 
      ($parameters !== null) ? $query = http_build_query($parameters) : $query = '';
 
      if($requestType === 'POST'){
 
            curl_setopt($ch,CURLOPT_POST, 1);                
            curl_setopt($ch,CURLOPT_POSTFIELDS, $query);  

      }elseif ($requestType === 'GET') {

            if($query) $url = $url . '?' . $query;

      }else{
 
            $response->success = false;
            $response->body = 'request type GET or POST is missing.';

            return $response;
      }
      
      curl_setopt($ch, CURLOPT_URL, $url);                  
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);       
      curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);      
 
      if ($headers !== null) curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

      $result = curl_exec($ch);                             

      if($result !== false){

            $response->success = true;
            $response->body = $result;

      }else{

            $response->success = false;
            $response->body = curl_error($ch);
            $response->error = curl_errno($ch);
      }                  
      
      curl_close($ch);                                      

      return $response;                                       
}
 
 
 
?>

 

Â