Redirecting when not in context of a controller

  $response = new RedirectResponse($url->toString());
  $request = \Drupal::request();
  // Save the session so things like messages get saved.
  $request->getSession()->save();
  $response->prepare($request);
  // Make sure to trigger kernel events.
  \Drupal::service('kernel')->terminate($request, $response);
  $response->send();

This does the job (almost), and one could simply wrap this in a new drupal_goto() function and call it a day.

But once caching is turned on this stops working as expected.

Testing the caching issue

To show the issue, I've added a redirect inside the THEME_preprocess_html():

function mytheme_preprocess_html(&$variables) {
  // Redirect when there's a "redirect-test" query parameter
  if (\Drupal::request()->query->has('redirect-test')) {
    // Prepare the goto Url.
    $url = Url::fromUserInput('/')
      ->setAbsolute()
      ->toString();

    // Create the response.
    $response = new TrustedRedirectResponse($url);
    // And add the url.query_args as the cache context.
    $response->getCacheableMetadata()->addCacheContexts(['url.query_args']);

    // Redirect code from https://www.drupal.org/node/2023537
    $request = \Drupal::request();
    $request->getSession()->save();
    $response->prepare($request);
    \Drupal::service('kernel')->terminate($request, $response);
    $response->send();
  }
}

Adding a ?redirect-test query string to the URL, will redirect the user to the frontpage (/).
This works the first time around, but the second time it simply shows the page and doesn't send any redirection.

Let's have a look at the headers (I've removed most of the headers, and only kept the ones relevant for this example):

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 302 Found
Cache-Control: no-cache, private
Location: https://example.com/

This first request sends the user the / location.

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 200 OK
Cache-Control: max-age=300, public
X-Drupal-Cache: HIT

The second request however shows the requested node (note the X-Drupal-Cache: HIT) and doesn't set the location.

On the second request the mytheme_preprocess_html is never reached (because the page is cached), and it never sends the redirect response.

Fixing redirection and caching

So the idea is to throw an exception, this exception is handled via an event_subscriber and it will change the response.

The implementation

I've compiled the files in a gist for a quick overview.

Create a new exception and an event subscriber, In any module (we've implemented the functionality in our agency general purpose module, but it can be anywhere).
In the example implementation below the module is called drupalgoto.

The exception is simple, the important thing is it needs a response, which is what the event subscriber will send to the client:

namespace Drupal\drupalgoto\EventSubscriber;

use Symfony\Component\HttpFoundation\Response;

/**
 * Override response exception.
 */
class OverrideResponseException extends \RuntimeException {

  /**
   *
   */
  protected $response;

  /**
   * Construct instance.
   */
  public function __construct(Response $response = null, ?string $message = '', \Throwable $previous = null, int $code = 0) {
    $this->response = $response;
    parent::__construct($message, $code, $previous);
  }

  /**
   * Get response.
   */
  public function getResponse() {
    return $this->response;
  }

}

Now the event subscriber should listen for this type of exception, and set the response to the response set in the exception:

namespace Drupal\drupalgoto\EventSubscriber;

use Drupal\drupalgoto\Exception\OverrideResponseException;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Symfony\Component\HttpKernel\Event\ExceptionEvent;
use Symfony\Component\HttpKernel\KernelEvents;

/**
 * Event subscriber.
 */
class EventSubscriber implements EventSubscriberInterface {

  /**
   * Exception event.
   */
  public function onKernelException(ExceptionEvent $event) {
    $throwable = $event->getThrowable();

    // Check if it's the OverrideResponseException.
    if ($throwable instanceof OverrideResponseException) {
      // And set the response if it is.
      $event->setResponse($throwable->getResponse());
    }
  }

  /**
   * {@inheritdoc}
   */
  public static function getSubscribedEvents() {
    return [
      KernelEvents::EXCEPTION => ['onKernelException'],
    ];
  }

}

The magic is inside the onKernelException() function, here it tests for the exception and then sets the response accordingly.
Remember to register the event subscriber in the drupalgoto.services.yml:

ervices:
  drupalgoto.event_subscriber:
    class: Drupal\drupalgoto\EventSubscriber\EventSubscriber
    tags:
      - { name: event_subscriber }

That's it. It's a lot of boilerplate and extra code, but now you're ready to throw the OverrideResponseException and redirect the user.

Using the implementation

Changing the previous test case to use the new exception throwing variant:

function mytheme_preprocess_html(&$variables) {
  // Redirect when there's a "redirect-test" query parameter
  if (\Drupal::request()->query->has('redirect-test')) {
    // Prepare the goto Url.
    $url = Url::fromUserInput('/')
      ->setAbsolute()
      ->toString();

    // Create the response.
    $response = new TrustedRedirectResponse($url);
    // And add the url.query_args as the cache context.
    $response->getCacheableMetadata()->addCacheContexts(['url.query_args']);

    // Throw the override exception:
    throw new OverrideResponseException($redirect);
  }
}

It's the same as the last test, except this throws the OverrideResponseException, instead of the approach from https://www.drupal.org/node/2023537.

Is it working?

Let's run the previous test from the terminal (I've stripped a lot of unimportant headers from the output again):

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 302 Found
Cache-Control: no-cache, private
Location: https://example.com/

The first run works just the same as with the old method.

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 200 OK
Cache-Control: max-age=300, public
X-Drupal-Cache: HIT
Location: https://example.com/

On the second call, notice the Location, so this time the redirect is cached and executed correctly.

Setting http.response.debug_cacheability_headers to TRUE on the Drupal installation will show caching information in the headers.
Doing this adds the X-Drupal-Cache-Contexts: url.query_args user.permissions header to both the first and the second call, telling us the url.query_args cache context is added as expected.

Conclusion

Even though Drupal 7 was a long time ago, I'll make the comparison because I've previously mentioned the drupal_goto() function.
So like anything in Drupal 8+ it's a lot more boilerplate and extra code compared to Drupal 7. But the extra code allows for an excellent caching system, so it's worth it!

And once you've implemented the override response system once (in its own module or a base module if you have one) it's always there ready to use.

Icing on the cake

We've implemented a wrapper function for even easier HTTP redirects, this makes it a simple one-liner with no need to reference the TrustedRedirectResponse class:

public static function goto(Url $url, $cache = []) {
  $response = TrustedRedirectResponse::create($url->setAbsolute()->toString())
    ->addCacheableDependency(CacheableMetadata::createFromRenderArray(['#cache' => $cache]));

  throw new static($response);
}

Notice this makes sure the URL is absolute, and calls throw. It also wraps the cache allowing for a simple array setting the cache.
A simple one-liner of the previous example:

OverrideResponseException::goto(Url::fromUserInput('/'), ['contexts' => ['user']]);

Implementation

It's a small proxy, with a [[ProxyHandler]] that traps [[Get]] and [[Call]], it simply returns the proxy object itself in each trap:

const neverendingObject = new Proxy(function () { }, {
  apply: function () { return neverendingObject; },
  get: function () { return neverendingObject; }
});

Note the [[ProxyTarget]] is a function, this will allow a neverendingObject() function call without any extra chaining.

For educational purposes, I'll add logging in the traps, this way it's easier to get what's happening:

const neverendingObject = new Proxy(function () { }, {
  apply: function (target) { console.log('[[Call]] ()'); return neverendingObject; },
  get: function (target, prop) { console.log('[[Get]]', prop); return neverendingObject; }
});

neverendingObject.aProperty.aFunction().aProperty

// [[Get]] aProperty
// [[Get]] aFunction
// [[Call]] ()
// [[Get]] aProperty

How it works

I had some trouble wrapping my head around the apply part, the extra get call getting the reference to aFunction first and then calling it after.
I've included the call chain below highlighting each step of the way, thinking of it that way helped me out (remember that each [[Get]] and [[Call]] will return the neverendingObject object).

1. neverendingObject [[Get]] aProperty

2. neverendingObject.aProperty [[Get]] aFunction

3. neverendingObject.aProperty.aFunction [[Call]] ()

4. neverendingObject.aProperty.aFunction() [[Get]] aProperty

5. neverendingObject.aProperty.aFunction().aProperty

Hopefully, this didn't add to any confusion.

The optional chaining (?.) case

The neverendingObject can not replace the ?. operator, but it's possible to mimic the behavior of chaining without testing every step. but only if the last action is a function, and yes that's a show-stopper for most use cases.

const person = {
  arms(hasNoArms) {
    if (hasNoArms === true) {
      return neverendingObject
    } 

    return {
      wave() { console.log('Waving arms'); }
    };
  }
}

person.arms().wave(); // "Waving arms"
person.arms(true).wave(); // Nothing happens

Using the ?. operator, it would look like the following (assuming null is returned instead of the neverendingObject, when the hasNoArms parameter is set):

person.arms(true)?.wave(); // Nothing happens

The ?. operator is more explicit and would be the preferred way to go most of the time. As a bonus the ?. operator stops execution of the rest of the chain, where the neverendingObject will need to run the chain to its end, including any expensive calculation as an argument.

An actual use case

With everything, there's wrong about the neverendingObject, I found a use case where it shines, which is adding conditional chaining to existing objects.

A very simple case is adding an and function to the console object, giving an easy way to only log output if a condition is true without the added if:

console.and = function (condition) {
  if (condition) {
    return console;
  }
  return neverendingObject;
}

console.and(1 == 1).log('is logged'); // "is logged"
console.and(1 == 2).log('is logged'); // Nothing happens

Conclusion

Let's compare a simple if with the and() function:

if (isDebugging) {
  console.log('Some message');
}

// vs

console.and(isDebugging).log('some message');

It saves a couple of lines, and it's (subjectively) easier to read. When using the time() and timeEnd() to do some light profiling those couple of lines add up:

if (isDebugging) {
  console.time('abcd');
}
// Expensive stuff
if (isDebugging) {
  console.timeEnd('abcd');
}

// vs

console.and(isDebugging).time('abcd');
// Expensive stuff
console.and(isDebugging).timeEnd('abcd');

I've started a new branch of an old project: ConditionalConsole.
The previous version needed to create another object than console. Using this technique I'm able to decorate it and make the functionality have a more native feel.

The Search API module is the de facto standard of adding search to a Drupal site. It supports many search backends and has a built-in internal Database backend, that is great for testing and smaller sites. It also has an impressive amount of index settings, field data types, processors, and the ability to integrate with facets.

These are all necessary to make a complete search experience, even creating a simple searcher requires you to manage (and understand) many of these options.

This makes it a daunting task to create a search index. I know I didn't feel comfortable, working with the Search API module at first, and have since revisited my first implementations to change the most glaring beginner mistakes.

One of the (my) biggest challenges is not knowing what entities had indexed what, and if they had been indexed.

Enter the Search API Generic Devel module

After installing the module, there should appear a new Search API tab under the Devel tab on an indexed node (or any entity that is being indexed by any Search API index).

Try it yourself

The screenshot above is from an Umami installation, where I've installed search_api_gendev and the search_api_db module.

You can use Simplytest to try what you see in this article yourself (or spin up your own installation).

Enable the search_api_db module and import the config from this gist.

Using the Index item and Delete item actions will do exactly what you would expect. Try them out and see that there's now content in the Indexed data container:

Not there yet

You'll notice that the Local data is updated immediately when altering the entity, which is to be expected, but you'll notice the Indexed data is also updated. This is because it simply loads the Item, and it doesn't fill it with data from the backend (the backends don't have a uniform way of returning this information), however, some backend implementations will include extraData, that could hold the data actually indexed.

The extraData

Depending on the backend server used there's be access to more information. Some backend implementations will add more data to the extraData property of the indexed item. The extraData is shown in the Indexed data container if any is provided by the backend server implementation (the Solr module will provide extra information).

Conclusion

Even with the shortcomings (of not being able to see exactly what every backend has actually indexed), I find the Search API Generic Devel module extremely useful. Getting a view of what the rendered HTML output is has helped me debug a lot faster than previously.

It also eases the onboarding of new developers that haven't worked with the Search API before, which is a huge win.

The idea

Is to create a ProxyStats class that can time all calls to an object. I'll walk through the class creation in iterations, revisiting methods as functionality is added.

Initial implementation

Browse the code at github (commit #05a5e66)

class ProxyStats {
  constructor() {
    this._timers = {};
    this.stats = {};
  }

  static watch(target) {
    const proxyStats = new ProxyStats();
    return proxyStats.watch(target);
  }

  watch(target) {
    return new Proxy(target, this)
  }

  getTimeKey(target, property) {
    return `${target.constructor.name}.${property}`;
  }

  time(key) {
    this._timers[key] = Date.now();
  }

  timeEnd(key) {
    const time = Date.now() - this._timers[key];
    this.updateStats(key, time);

    console.log(key, `${time} ms (${this.stats[key].time} ms -- ${this.stats[key].count})`);
  }

  updateStats(key, time) {
    this.stats[key] = this.stats[key] || {time: 0, count: 0};

    this.stats[key].time +=  time;
    this.stats[key].count += 1;
  }

  get(target, property, receiver) {
    const timeKey = 'get ' + this.getTimeKey(target, property);

    this.time(timeKey);
    const result = Reflect.get(target, property, receiver);
    this.timeEnd(timeKey);

    return result;
  }

  set(target, property, value, receiver) {
    const timeKey = 'set ' + this.getTimeKey(target, property);

    this.time(timeKey);
    Reflect.set(target, property, value, receiver);
    this.timeEnd(timeKey);

    return true;
  }
}

The getTimeKey(), time(), timeEnd() and updateState() is used for timing and counting the calls to the object.

Both the static and prototype watch() function is used for watching an object.
The static function will automatically create a new ProxyStats, while the prototype function can be used to watch several objects with the same handler instance.
Using the same ProxyStats instance to watch several objects is helpful if you want to get statistics across all objects watched.

The get()^spec and set()^spec traps are both simple implementations. Using the Reflect^spec object to set or get the original value. This call is timed, and information is added to the ProxyStats.stats object.

Test setup

Prepare a simple class to test the proxy handler. Notice the 50 ms delays added in the getter, setter and function.

function slowFunction(min, max) {
  const end = Date.now() + Math.round(min + Math.random() * (max - min));
  while (Date.now() < end) { }
}

class ExampleClass {
  aProperty = 'some property'

  get aGetter() {
    slowFunction(50, 50);
    return 'some getter';
  }

  set aGetter($value) {
    slowFunction(50, 50);
  }

  aFunction() {
    slowFunction(50, 50);
  }
}

Now wrap the ExampleClass using the watch() function, and calling the different properties to see the output.

const exampleObject = ProxyStats.watch(new ExampleClass());
exampleObject.aProperty;
exampleObject.aGetter;
exampleObject.aGetter = '';
exampleObject.aFunction();

Output

The output here is the console output written in the timeEnd() function, it's the following format: KEY TIME ms (TOTAL_TIME ms -- COUNT).

• KEY: The key tells what kind of trap is used, and the object property.
• TIME: Is the time of the current call.
• TOTAL_TIME: Time of all calls for that specific property.
• COUNT: The amount of times the property has been called.

get ExampleClass.aProperty 0 ms (0 ms -- 1)
get ExampleClass.aGetter 50 ms (50 ms -- 1)
get ExampleClass.aGetter 50 ms (100 ms -- 2)
set ExampleClass.aGetter 50 ms (50 ms -- 1)
get ExampleClass.aFunction 0 ms (0 ms -- 1)
get ExampleClass.aFunction 0 ms (0 ms -- 2)

Getting a property (get ExampleClass.aProperty) is instant, so this seems fine at 0 ms.

Both the getter (get ExampleClass.aGetter) and setter (set ExampleClass.aGetter) takes 50 ms as expected.

But the function call (get ExampleClass.aFunction) is also instant, which is not correct. The reason it's instant is it only gets the property value, which is a function, it's not executing it.

Getting function timing to work

Browse the code at github (commit #bb97f3f)

To time function calls, the ProxyStats needs to wrap each function in another Proxy that utilizes the apply()^spec trap.

This is done by detecting functions in the get() trap and returning a new Proxy, created by the helper function handleFunction():

handleFunction(target, property, receiver) {
  const me = this;

  return new Proxy(target[property], {
    apply(fnTarget, thisArg, args) {
      const timeKey = me.getTimeKey(target, property) + '()';
      me.time(timeKey);
      const result = Reflect.apply(fnTarget, thisArg, args);
      me.timeEnd(timeKey);

      return result;
    }
  });
}

get(target, property, receiver) {
  if (typeof target[property] === 'function') {
    return this.handleFunction(target, property, receiver);
  }

  const timeKey = 'get ' + this.getTimeKey(target, property);

  this.time(timeKey);
  const result = Reflect.get(target, property, receiver);
  this.timeEnd(timeKey);

  return result;
}

It's a tiny rewrite of get() that returns early if the property is a function. All the functionality is in the handleFunction() function

Note that the new Proxy wraps the actual function target[property], and timing is simple and done the same way as with properties and getters.

Recursive functions

Browse the code at github (commit #c652f33)

The last issue in the POC is recursive functions. When the recursive function calls itself, it resets the timer because the key is identical. Updating the example object will give a clear view of the problem. So a recursive function is added to the ExampleClass.

aRecursiveFunction(depth) {
  slowFunction(50, 50);

  if (--depth > 0) {
    this.aRecursiveFunction(depth);
  }
}

Calling this function with depth 2 exampleObject.aRecursiveFunction(2); will give the following output:

ExampleClass.aRecursiveFunction() 0 ms (0 ms -- 1)
ExampleClass.aRecursiveFunction() 0 ms (0 ms -- 2)
ExampleClass.aRecursiveFunction() 0 ms (0 ms -- 3)

To fix this the timer key is made unique, so it no longer resets every time a function with the same name is called. This is done in the handleFunction() function by adding a random string to the key:

handleFunction(target, property, receiver) {
  const me = this;

  return new Proxy(target[property], {
    apply(fnTarget, thisArg, args) {
      const timeKey = me.getTimeKey(target, property) + '() -- ' + Math.random().toString(36).substring(2);
      me.time(timeKey);
      const result = Reflect.apply(fnTarget, thisArg, args);
      me.timeEnd(timeKey);

      return result;
    }
  });
}

The important part here is the random string suffix added to the timeKey.
Don't get confused about the Math.random().toString(36).substring(2) it's just a clever way of making a random string, try running it a couple of times in the console. This trick is purely cosmetics a Math.random() would've done the job.

This creates a problem with aggregating function calls, and makes it impossible, to sum up totals, check out the output:

ExampleClass.aFunction() -- mq55zfo3g99 50 ms (50 ms -- 1)
ExampleClass.aFunction() -- hw42ikytz5m 50 ms (50 ms -- 1)
ExampleClass.aRecursiveFunction() -- sirt14hqvn 50 ms (50 ms -- 1)
ExampleClass.aRecursiveFunction() -- hfcibln8qjf 100 ms (100 ms -- 1)

To solve this the suffix is removed from the key before calling updateStats() in the timeEnd(). This way the key used in the _timers is unique, but key used in the stats is only the function name without the unique part:

timeEnd(key) {
  const time = Date.now() - this._timers[key];

  key = key.split(' -- ').shift();
  this.updateStats(key, time);

  console.log(key, `${time} ms (${this.stats[key].time} ms -- ${this.stats[key].count})`);
}

This gives a result without the unique part:

ExampleClass.aFunction() 50 ms (50 ms -- 1)
ExampleClass.aFunction() 50 ms (100 ms -- 2)
ExampleClass.aRecursiveFunction() 50 ms (50 ms -- 1)
ExampleClass.aRecursiveFunction() 100 ms (150 ms -- 2)

There's still an issue with the total sum of the recursive function.

• Initial call waits 50 ms, then calls itself.
• The second call waits 50 ms before logging the time and ending the second call.
• Now the initial call has been 100 ms underway, this is logged and initial call ended.
• The total sum is calculated as second call + initial call, which is 150 ms even though only 100 ms is actually used in total.

At this time it's good enough for jazz, but this issue will be fixed later on.

Future

So far so good, the proof of concept seems to be working as intended (except for recursive functions).

As I'll be expanding on the ProxyStats class I'll try to create relevant blog posts. The following is a list of possible topics (also functioning as notes for myself, so every bullet might not justify a blog post), in no particular order:

• Fix the recursive function issue.
• Documenting the class, using JSDoc.
• How to use the ProxyStats class.
• Testing, with simple unit tests.
• Adding more features, filtering calls, smarter watcher functions.
• Iterating existing functions and reviewing/rewriting code.
• Performance, any obvious performance improvements.

Echo message

The ICMP header starts after the regular IP header (which is handled by socket_connect). It's a pretty simple header of 8 bytes, followed by the message being sent.

╔═══════════════╦═══════════════╦══════════════════════════════╗
║    Type 8b    ║    Code 8b    ║         Checksum 16b         ║
╠═══════════════╩═══════════════╬══════════════════════════════╣
║         Identifier 16b        ║      Sequence number 16b     ║
╠═══════════════════════════════╩══════════════════════════════╣
║                     data (variable length)                   ║
╚══════════════════════════════════════════════════════════════╝

• Type: The type of control message to send. The echo message is 8, but there are a lot of other types, for other commands.
• Code: An extra setting for the type, depending on the type. The echo message doesn't have any extra settings, so it's left at 0.
• Checksum: This is calculated after the package is assembled, to start with it's simply set to 0. The package is paired into 16-bits integers, and one's complement of the sum(explained later on).
• Identifier: The request identifier can be anything, it's used to match the reply. In the original ping, this was the UNIX process ID, but we'll leave it 0.
• Sequence number: Like the identifier, this is used to match the reply, left at 0. The original ping increments this on every ping.
• Data: The message. Like with the Identifier and Sequence number, the destination needs to reply with the same values. Original ping sends a timeval struct to compute the round-trip.

Calculating the checksum

In this example, the package used is phping (when sending a ping the package will be the entire header and the data to send)

1) Divide the package into 16-bit pairs.

The package phping is the following in binary:

p: 0111 0000
h: 0110 1000
p: 0111 0000
i: 0110 1001
n: 0110 1110
g: 0110 0111

This is paired into 16-bit integers:

a: 0111 0000 0110 1000
b: 0111 0000 0110 1001
c: 0110 1110 0110 0111

2) Add all the pairs to get the sum.

Adding a, b and c give an integer larger than 16 bit:

a + b + c: 1 0100 1111 0011 1000

3) Sum 16-bit pairs of the sum again.

Reiterate the first 2 steps, until the sum is a single 16-bit integer.

Divide it:

a: 0000 0000 0000 0001
b: 0100 1111 0011 1000

Sum it:

a + b: 0100 1111 0011 1001

4) End with one's complement, to invert the integer.

One's complement is the same as a bitwise NOT operation.

~ 0100 1111 0011 1001: 1011 0000 1100 0110

PHP checksum implementation

There are a lot of tips and tricks, in the previously mentioned RFC 1071 for implementing this (and some examples). This is just one way of doing it.

function computeInternetChecksum($in) {
  // Add an empty char (8-bit).
  // This trick leverages the way unpack() works.
  $in .= "\x0";

  // The n* format splits up the data string into 16-bit pairs.
  // It will unpack the string from the beginning, and only split
  // whole pairs. So it will automatically leave out (or include) the
  // odd byte added above.
  $pairs = unpack('n*', $in);

  // Sum the pairs.
  $sum = array_sum($pairs);

  // Add the hi 16 to the low 16 bits, ending in a single 16-bit int.
  while ($sum >> 16)
    $sum = ($sum >> 16) + ($sum & 0xffff);

  // End with one's complement, to invert the integer.
  // Note the ~ operator before packing the sum into a string again.
  return pack('n', ~$sum);
}

Check phping gives the expected checksum:

$checksum = computeInternetChecksum('phping');

// Note that unpack() returns an array starting 1 (not 0).
echo decbin(unpack('n*', $checksum)[1]);

// Output (the same as manually calculated):
// 1011000011000110

If you're interested check out the original ping checksum implementation.

Another PHP implementation, that resembles the original ping implementation (and C implementation in the RFC) can be found in the socket_create() comments.

Preparing the package

The package is set up following the ICMP header schema:

// Prepare the package.
$package = [
  'type' => "\x08",
  'code' => "\x00",
  'checksum' => "\x00\x00",
  'identifier' => "\x00\x00",
  'seqNumber' => "\x00\x00",
  'data' => 'phping',
];

// Compute the checksum, so it's ready to send.
$package['checksum'] = computeInternetChecksum(implode('', $package));
$rawPackage = implode('', $package);

Check the package is as expected:

// Unpack the package into an associated array.
$icmpHeaderFormat = 'Ctype/Ccode/nchecksum/nidentifier/nsequence';
// And show the binary values of each header part.
print_r(array_map('decbin', unpack($icmpHeaderFormat, $rawPackage)));

// Output:
// Array
// (
//     [type] => 1000
//     [code] => 0
//     [checksum] => 1010100011000110
//     [identifier] => 0
//     [sequence] => 0
// )

Moving on to sockets

Sending the package is done by using the PHP socket functions.

ICMP requests need raw network access, this causes permission issues, more on this later.

// Create the socket.
// AF_INIT is the IPv4 protocol.
// SOCK_RAW is needed to perform ICMP requests.
$socket = socket_create(AF_INET, SOCK_RAW, getprotobyname('icmp'));

// Open up the connection to a host.
socket_connect($socket, 'google.com', null);

// Used to calculate the response time.
$time = microtime(true);

// Send the package to the target host.
socket_send($socket, $rawPackage, strlen($rawPackage), 0);

// Read the response.
if ($in = socket_read($socket, 1)) {
  // Print the response time.
  echo microtime(true) - $time . " seconds\n";
}

// Close the socket.
socket_close($socket);

This doesn't check the reply, and simply assumes any reply is valid. This is also why only 1 byte is read in the socket_read.

To check the reply, you would need to parse the input (including the IPv4 header) and verify the checksum.

The SOCK_RAW issue.

Because of security risks, root / administrator access is needed to use SOCK_RAW (or the PHP executable needs CAP_NET_RAW capability).

So when testing the script you'll need to sudo the command (I believe the Windows equivalent would be run as administrator).

PHP will trigger the following warning (this may vary depending on OS), if it's not run with sufficient permissions:

PHP Warning:  socket_create(): Unable to create socket [1]: Operation not permitted

Conclusion

First the script, ready for copy paste:

<?php

function computeInternetChecksum($in) {
  // Add an empty char (8-bit).
  // This trick leverages the way unpack() works.
  $in .= "\x0";

  // The n* format splits up the data string into 16-bit pairs.
  // It will unpack the string from the beginning, and only split
  // whole pairs. So it will automatically leave out (or include) the
  // odd byte added above.
  $pairs = unpack('n*', $in);

  // Sum the pairs.
  $sum = array_sum($pairs);

  // Add the hi 16 to the low 16 bits, ending in a single 16-bit int.
  while ($sum >> 16)
    $sum = ($sum >> 16) + ($sum & 0xffff);

  // End with one's complement, to invert the integer.
  // Note the ~ operator before packing the sum into a string again.
  return pack('n', ~$sum);
}

// Prepare the package.
$package = [
  'type' => "\x08",
  'code' => "\x00",
  'checksum' => "\x00\x00",
  'identifier' => "\x00\x00",
  'seqNumber' => "\x00\x00",
  'data' => 'phping',
];

// Compute the checksum, so it's ready to send.
$package['checksum'] = computeInternetChecksum(implode('', $package));
$rawPackage = implode('', $package);

// Create the socket.
// AF_INIT is the IPv4 protocol.
// SOCK_RAW is needed to perform ICMP requests.
$socket = socket_create(AF_INET, SOCK_RAW, getprotobyname('icmp'));

// Open up the connection to a host.
socket_connect($socket, 'google.com', null);

// Used to calculate the response time.
$time = microtime(true);

// Send the package to the target host.
socket_send($socket, $rawPackage, strlen($rawPackage), 0);

// Read the response.
if ($in = socket_read($socket, 1)) {
  // Print the response time.
  echo microtime(true) - $time . " seconds\n";
}

// Close the socket.
socket_close($socket);

And the result:

$ sudo php ping.php
0.015023946762085 seconds

Because of the SOCK_RAW limitation, it's mostly an exercise in working with sockets, RFC documentations and binary string handling in PHP.

It might have its merits in a PHP CLI script, or if the PHP executable called by the webserver can get the CAP_NET_RAW using setcap.

If you have a blog hosted on hashnode you'll automatically get a lot of great features. One of these essential blog features is the RSS feed, allowing readers to subscribe with whatever RSS reader they are most comfortable with.

The problem is this feed is for every post, so if you have a blog with many different topics (different programming languages, frameworks, personal posts, movie reviews) your readers can't subscribe to a single topic but will receive all your posts. Or maybe you want to add your feed to a planet or another aggregator with a specific topic (my own usecase).

The solution

An externally hosted script, that parses the RSS and returns only the relevant posts. A reader (or feed aggregator) could then subscribe to this feed instead, and only get the posts they're interested in.

The stack

Code is available at the following GitHub repo (the readme holds a lot of the information in this blog post): https://github.com/BirkAndMe/hashnoderss

The stack for this project is kept as simple as possible, it only consists of PHP. One just needs a host with PHP support, this can be any shared host, dedicated server or cloud solution.

It's a standalone script, with no ties to any external libraries. And since the script is as small as it is, everyone with PHP knowledge should be able to grasp what's going on and understand the ins and outs, no matter what framework they usually work with. Making it more available to everyone.

The installation

Testing the code locally.

_There's also a quick intro to testing locally in the repository.

This is a quick and dirty way of testing out the functionality, and should not be used for any actual production.

git clone git@github.com:BirkAndMe/hashnoderss.git
cd hashnoderss
php -S localhost:8000 router-hack.php

Quick intro to a Linode LEMP installation.

Use whatever hosting option you're comfortable with, this is just a quick getting started.

Since the script doesn't use MySQL you could opt for manually setting up an instance with just a web server and PHP, but that is out of scope for this post

1) Find and install LEMP in the Linode marketplace.

Fill in all the options, and start by choosing the smallest (and cheapest) Shared CPU (Nanode 1 GB at the time of writing). The instance should start automatically and setup everything for a standard LEMP setup.

1.1) Open an SSH connection to the Linode instance.

Or use the Linode LISH console if you prefer.

2) Getting the repository into the web root.

Find the webroot (on the Linode instance) in /var/www/html/[SITE_NAME]/public_html and install the script here (notice git is installed, and the directory is cleaned).

sudo apt install git
rm *
git clone https://github.com/BirkAndMe/hashnoderss.git .

The [SITE_NAME] is the Reverse DNS name of the Linode instance (should be something like [IP-ADDRESS].ip.linodeusercontent.com). You can find this name in the Network tab when inspecting your Linode instance in the Linode backend https://cloud.linode.com/linodes/[INSTANCE_ID]/networking.

3) Setting up PHP and NGINX.

Just a few changes are needed to host the script.

3.1) Install PHP XML.

The XML module is not default for the PHP Debian installation, so install it.

sudo apt install php-xml

3.2) Change the NGINX site config.

Edit the site configuration found in /etc/nginx/sites-enabled/[SITE_NAME], on a clean installation there should only be 1 site so that's it (the server has nano ready for editing, but use whatever you like).

Setup a redirect of all the requests to the index.php script. Do so by replacing this:

location / {
  try_files $uri $uri/ =404;
}

With this:

location / {
  try_files $uri $uri/ /index.php;
}

Reload NGINX by running this command

systemctl reload nginx`

Now visit the site to make sure everything works (this should get the complete RSS feed for this blog): https://[SERVER-IP].ip.linodeusercontent.com/blog.birk-jensen.dk

The API

The primary (and only) endpoint is:

https://host/[BLOG_DOMAIN]

It's possible to set a blog domain in the settings.php (as $hostname). Doing this will ignore the BLOG_DOMAIN, and using https://host will work without the extra path.

Filtering

This is filtered using Query parameters.

These parameters are mapped to PostDetailed query on api.hashnode.com, so use the documentation available there to figure out what properties to filter by.

Simple filter.

host/blog.birk-jensen.dk?_id=6298c0135787a911a45d5fcf

This will get a single blog post with a specific ID.

Nested filtering.

It's possible to check nested properties, using double dashes -- as a seperator, so the following is probably the most useful filter:

host/blog.birk-jensen.dk?tags--name=Drupal

Only return blog posts tagged with Drupal.

Combining filters.

host/blog.birk-jensen.dk?author--username=BirkAndMe&tags--name[]=Drupal

Get blog posts written by BirkAndMe and tagged with Drupal.

The filter only support AND conditions.

Operators.

The value checks support 4 operators, which is written as a prefix (remember to URL escape) to the value

• = Equal, is the default operator and makes sure the post value is the same as the given value.
• ! Not, make sure the value is not the same.
• < Lesser than, only if the value is lesser than.
• > Greater than, only if the value is greater than.

Get all posts except a specific one.

host/blog.birk-jensen.dk?_id=!6298c0135787a911a45d5fcf

Only posts with more than 5 reactions (the %3E is > escaped).

host/blog.birk-jensen.dk?totalReactions=%3E5

Debugging.

Add a debug parameter to get a better view of what's going on. It will show the normalized values and the Graph QL used to query hashnode.

host/blog.birk-jensen.dk?author--username=BirkAndMe&tags--name[]=Drupal&debug

The code

Repo link again: https://github.com/BirkAndMe/hashnoderss

There's not much to be said about the code, it should be pretty self-explanatory.

It's kept simple rather than smart. Whenever I could do something smart, I chose to do something explicit instead, to make the code more approachable.

Because of the simplicity, reading the index.php from the beginning should give a pretty good picture of what's going on.

Here's the gist of it (gives you an overview, before diving into the code):

1. Read the original RSS feed from the blog.
2. Get all URLs in the original RSS (these are used as slugs when querying the Hashnode api.
3. Normalize the query parameters into a filter array, and use this array to build the Graph QL property list.
4. Query the Hashnode API, and run through all the posts checking the filter matches (and removing the posts that don't match from the RSS).
5. Return the filtered RSS.

Remarks

I hope anyone other than me can use this for their Hashnode blog (I might set up an actual service in the future, so you don't have to host the script).

I also someone will find it refreshing to read a simple not framework dependant open source project. I know I found it refreshing to write something using only the simplest of tools.

Real-life use case.

I haven't tested this in production yet, but I would likely set up an HTTP cache (varnish) in front of this to minimize all the external requests (RSS and API calls), this should be almost plug and play, since it only uses GET requests.

Also using the $hostname setting and then having the RSS feed on a rss.domain.com will give prettier URLs.

Future features.

My guess is that tag filtering is the only needed filter 99.9% of the time, so any future feature is primary just for the fun of it, but here goes:

• Get the greater and lesser than working on dates.
• Implement a regular expression operator.
• Improve code documentation on helper functions.
• Error handling (there's none at the moment).
• Get posts from a specific user (using the user query in the Hashnode API) across multiple blogs.

Limitations.

The Hashnode API is slow, especially when querying many posts (to the point where it sometimes fails). This is why it limits the posts to a max of 20, in most cases this is more than enough because RSS clients will only look for new responses anyway.

It also caches the requests for 10 minutes, so you won't get anything instantly.