The drupal_goto() function was removed in Drupal 8, and this is all good!
In the change record there's an example, of how to do a redirection outside the context of a controller:

Redirecting when not in context of a controller

  $response = new RedirectResponse($url->toString());
  $request = \Drupal::request();
  // Save the session so things like messages get saved.
  $request->getSession()->save();
  $response->prepare($request);
  // Make sure to trigger kernel events.
  \Drupal::service('kernel')->terminate($request, $response);
  $response->send();

This does the job (almost), and one could simply wrap this in a new drupal_goto() function and call it a day.

But once caching is turned on this stops working as expected.

Testing the caching issue

To show the issue, I've added a redirect inside the THEME_preprocess_html():

function mytheme_preprocess_html(&$variables) {
  // Redirect when there's a "redirect-test" query parameter
  if (\Drupal::request()->query->has('redirect-test')) {
    // Prepare the goto Url.
    $url = Url::fromUserInput('/')
      ->setAbsolute()
      ->toString();

    // Create the response.
    $response = new TrustedRedirectResponse($url);
    // And add the url.query_args as the cache context.
    $response->getCacheableMetadata()->addCacheContexts(['url.query_args']);

    // Redirect code from https://www.drupal.org/node/2023537
    $request = \Drupal::request();
    $request->getSession()->save();
    $response->prepare($request);
    \Drupal::service('kernel')->terminate($request, $response);
    $response->send();
  }
}

Adding a ?redirect-test query string to the URL, will redirect the user to the frontpage (/).
This works the first time around, but the second time it simply shows the page and doesn't send any redirection.

Let's have a look at the headers (I've removed most of the headers, and only kept the ones relevant for this example):

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 302 Found
Cache-Control: no-cache, private
Location: https://example.com/

This first request sends the user the / location.

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 200 OK
Cache-Control: max-age=300, public
X-Drupal-Cache: HIT

The second request however shows the requested node (note the X-Drupal-Cache: HIT) and doesn't set the location.

On the second request the mytheme_preprocess_html is never reached (because the page is cached), and it never sends the redirect response.

Fixing redirection and caching

So the idea is to throw an exception, this exception is handled via an event_subscriber and it will change the response.

The implementation

I've compiled the files in a gist for a quick overview.

Create a new exception and an event subscriber, In any module (we've implemented the functionality in our agency general purpose module, but it can be anywhere).
In the example implementation below the module is called drupalgoto.

The exception is simple, the important thing is it needs a response, which is what the event subscriber will send to the client:

namespace Drupal\drupalgoto\EventSubscriber;

use Symfony\Component\HttpFoundation\Response;

/**
 * Override response exception.
 */
class OverrideResponseException extends \RuntimeException {

  /**
   *
   */
  protected $response;

  /**
   * Construct instance.
   */
  public function __construct(Response $response = null, ?string $message = '', \Throwable $previous = null, int $code = 0) {
    $this->response = $response;
    parent::__construct($message, $code, $previous);
  }

  /**
   * Get response.
   */
  public function getResponse() {
    return $this->response;
  }

}

Now the event subscriber should listen for this type of exception, and set the response to the response set in the exception:

namespace Drupal\drupalgoto\EventSubscriber;

use Drupal\drupalgoto\Exception\OverrideResponseException;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Symfony\Component\HttpKernel\Event\ExceptionEvent;
use Symfony\Component\HttpKernel\KernelEvents;

/**
 * Event subscriber.
 */
class EventSubscriber implements EventSubscriberInterface {

  /**
   * Exception event.
   */
  public function onKernelException(ExceptionEvent $event) {
    $throwable = $event->getThrowable();

    // Check if it's the OverrideResponseException.
    if ($throwable instanceof OverrideResponseException) {
      // And set the response if it is.
      $event->setResponse($throwable->getResponse());
    }
  }

  /**
   * {@inheritdoc}
   */
  public static function getSubscribedEvents() {
    return [
      KernelEvents::EXCEPTION => ['onKernelException'],
    ];
  }

}

The magic is inside the onKernelException() function, here it tests for the exception and then sets the response accordingly.
Remember to register the event subscriber in the drupalgoto.services.yml:

ervices:
  drupalgoto.event_subscriber:
    class: Drupal\drupalgoto\EventSubscriber\EventSubscriber
    tags:
      - { name: event_subscriber }

That's it. It's a lot of boilerplate and extra code, but now you're ready to throw the OverrideResponseException and redirect the user.

Using the implementation

Changing the previous test case to use the new exception throwing variant:

function mytheme_preprocess_html(&$variables) {
  // Redirect when there's a "redirect-test" query parameter
  if (\Drupal::request()->query->has('redirect-test')) {
    // Prepare the goto Url.
    $url = Url::fromUserInput('/')
      ->setAbsolute()
      ->toString();

    // Create the response.
    $response = new TrustedRedirectResponse($url);
    // And add the url.query_args as the cache context.
    $response->getCacheableMetadata()->addCacheContexts(['url.query_args']);

    // Throw the override exception:
    throw new OverrideResponseException($redirect);
  }
}

It's the same as the last test, except this throws the OverrideResponseException, instead of the approach from https://www.drupal.org/node/2023537.

Is it working?

Let's run the previous test from the terminal (I've stripped a lot of unimportant headers from the output again):

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 302 Found
Cache-Control: no-cache, private
Location: https://example.com/

The first run works just the same as with the old method.

$ curl -D /dev/stdout -o /dev/null -s https://example.com/node/2?redirect-test
HTTP/1.1 200 OK
Cache-Control: max-age=300, public
X-Drupal-Cache: HIT
Location: https://example.com/

On the second call, notice the Location, so this time the redirect is cached and executed correctly.

Setting http.response.debug_cacheability_headers to TRUE on the Drupal installation will show caching information in the headers.
Doing this adds the X-Drupal-Cache-Contexts: url.query_args user.permissions header to both the first and the second call, telling us the url.query_args cache context is added as expected.

Conclusion

Even though Drupal 7 was a long time ago, I'll make the comparison because I've previously mentioned the drupal_goto() function.
So like anything in Drupal 8+ it's a lot more boilerplate and extra code compared to Drupal 7. But the extra code allows for an excellent caching system, so it's worth it!

And once you've implemented the override response system once (in its own module or a base module if you have one) it's always there ready to use.

Icing on the cake

We've implemented a wrapper function for even easier HTTP redirects, this makes it a simple one-liner with no need to reference the TrustedRedirectResponse class:

public static function goto(Url $url, $cache = []) {
  $response = TrustedRedirectResponse::create($url->setAbsolute()->toString())
    ->addCacheableDependency(CacheableMetadata::createFromRenderArray(['#cache' => $cache]));

  throw new static($response);
}

Notice this makes sure the URL is absolute, and calls throw. It also wraps the cache allowing for a simple array setting the cache.
A simple one-liner of the previous example:

OverrideResponseException::goto(Url::fromUserInput('/'), ['contexts' => ['user']]);