Het is ons allen niet ontgaan vandaag: de enorme uitval van Facebook en haar dochterbedrijven zoals WhatsApp, Instagram en Messenger. Dit is niet zoals eerdere storingen die het platform had die beperkt waren tot bepaalde regio’s of diensten, en na enkele minuten verholpen waren.

Dit omdat de oorzaak van de storing van vandaag veel complexer is dan eerdere storingen die vermoedelijk “software bugs” waren die snel en gemakkelijk gecorrigeerd kunnen worden binnen het wereldwijde platform.

In deze blogpost probeer ik op een versimpeld te toe te lichten wat er vandaag is gebeurd, de impact hiervan, en waarom dit niet snel opgelost kan worden.

Of beluister mijn interview op Paradise FM over dit onderwerp:

I've been asked by Paradise FM, a local Dutch radio station, to share my expertise and thoughts on the recent development surrounding the (possible) intrusion that took place at the Central Bank of Curaçao and St. Maarten (CBCS). According to the latest press release at the time of writing/the interview, the CBCS claims that a possible intrusion took place through a remote work platform, possibly via a VPN (or similar tunneling) solution). In the interview I gave a brief introduction on VPN's and shared my initial thoughts about possible vulnrabilities exploited:

(interview is in Dutch)

So I recently found out the hard way that reverting back to an earlier/older EdgeRouter firmware image also reverts back to the configuration at the time of upgrade to the newer firmware. I expected the current (latest) configuration to remain active regardless of which firmware version you're running. In my case the router was upgraded to the 2.0.x firmware in our lab, before all production configuration was configured. When troubleshooting an issue I decided to downgrade the firmware just to confirm that this wasn't a result of a possible bug in the 2.0.x firmware. After reverting to the 1.10.x firmware I lost remote access to the router. When connecting to the console port I managed to log in and concluded that the settings ware as they ware at the time of the upgrade to the 2.0.x firmware. Luckily I made a back-up before reverting to the previous firmware so I was able to recover to the production configuration quickly.

username@router:~$ set system image default-boot 
The system currently has the following image(s) installed:

v2.0.9-hotfix.2.5402463.210511.1317 (running image) (default boot) 
v1.10.7.5127989.181001.1228    

Are you sure you want to switch images? (Yes/No) [Yes]: Yes
Moving images... 
Done
Switched from
  Version:      v2.0.9-hotfix.2.5402463.210511.1317
to
  Version:      v1.10.7.5127989.181001.1228
username@router:~$ reboot
Proceed with reboot? [confirm][y]

I thought this might be a word of caution to my peers :)

Last Sunday evening I published an extensive report on Flow Curaçao's internet service. At the time of writing I, as many others (if not for all Flow hybrid fiber customers), were having intermittent speed issues.

Once published the article was immediately picked up by Flow/CWC employes and immediately contacted me on the matter, requesting more in depth info, including subscriber information so they can investigate my findings and narrow down the problem.

I have to say that I am very pleased with the level of commitment shown from the Flow/CWC team to get to the bottom of the issue.

Therefor I am excited to announce that yesterday (June 15^th) the issue was identified and resolved. The issue was a defective interface on a 3rd party provider’s core network.

I can confirm that indeed the issue hasn’t returned since around midday yesterday:

The above graph data are the combined datapoints from June 15^th. In the graph it’s clear that since around 12pm no speeds below 1Mbit/s have been measured, indicating that the previously identified issue has not been observed.
(please ignore the difference in top speeds before/after around 14.30pm as I configure QoS rules to limit the impact on our bandwidth utilization)

A big thank you to the Flow/CWC team for their dedication. And if you’re a Flow internet customer you owe me a beer!

It shouldn’t have gone unnoticed to, the continuous complaints about Flow’s internet service in the press and on social media. I’ve noticed this as well. Before COVID the service was OK (as in usable), but during the COVID lockdown I began experiencing degraded speeds to (my own) servers (abroad) and other internet applications. Unfortunately, this trend has continued over the past months. Ultimately to the point where the service is almost unusable at times.

I wish this was an exception, but it isn’t. Because of moments like these the service becomes unusable…

This frustrated me as a “user” (client of the service/company) and an IT professional providing digital services where users are experiencing degraded performance, hence resulting in a degraded experience of my “service”.

To the point that I moved around one application from one service provider in Miami to another to get better “peering” (connectivity) with Flow’s network. This mitigated performance during the first lockdown but at this time we’re experiencing the same, well, far worse issues than we did previously.

This comes at a cost: both in hosting costs, labor (long nights migrating applications between hosts) and economic cost of not being able to deliver a good service to our end users (loosing revenue).

Aside from “cloud” services that we provide, we also spent many unnecessary hours trying to troubleshoot issues at (Sincere ICT’s) customers, having to prove that stability issues of the services they consume weren’t a result of internal network issues. Therefor assuming, because I couldn’t prove at the time, that these issues were a result of Flow’s service.

The economic impact of these issues is enormous, this makes businesses in Curaçao uncompetitive compared to the rest of the region or the world…

Hi there!

It was long overdue for me to revive my blog so I can share my thoughts, findings and projects I'm working on (personally). I won't write juicy articles frequently, but there's a juicy one in the making now. Stay tuned...