All systems operational

Server
Operational
Janus
99.92% uptime
Sep 2023 · 99.75%Oct · 100.0%Nov · 100.0%
Sep 202399.75% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Merkur

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Minerva

99.97% uptime
Sep 2023 · 100.0%Oct · 99.98%Nov · 99.94%
Sep 2023100.0% uptime
Oct 202399.98% uptime
Nov 202399.94% uptime

Services

99.65% uptime

Single Sign On

Operational

Notice history

Nov 2023

Minerva is down
  • Resolved
    Resolved

    We have identified the issue coming from our nginx reverse proxy running on Minerva as it seemed to be failing. In the logs we found nginx complaining about the "server names hash bucket size". More on that in this gist. The reboot of our system led to nginx performing a reload after a config change that lead to the error. We increased the bucket size as suggested and got nginx back up again. Our monitoring tools suggest that everything is back online again and we confirmed that. Thanks for your patience and sorry for any inconveniance caused.

  • Investigating
    Investigating

    We are currently investigating downtime of one of our servers after a sheduled reboot. Our monitoring systems in place are informing us about downtime since the reboot so it is very likely linked to that. We'll udate this incident once we know more.

Server - Minerva isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Server - Minerva is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Server - Minerva cannot be accessed at the moment. This incident was created by an automated monitoring service.

Oct 2023

Issues regarding account creation and password resets
  • Resolved
    Resolved

    We have confirmed the update resolved all issues. Functionality has been restored and everything is operational again. Thanks for your patience

  • Monitoring
    Monitoring

    The fix has been released and deployed. We are monitoring the results and will be re-enabling all features once we confirm the issue has been resolved.

  • Identified
    Identified

    We are aware of an issue affecting our single sign-on service that prevents the software from sending emails. Since this is required for both password resets and account creation, these options are currently unavailable.
    We are waiting for a patch from the software developer, which is expected to be available tomorrow at 19:00 UTC.

Server - Minerva isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Server - Minerva is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Server - Minerva cannot be accessed at the moment. This incident was created by an automated monitoring service.

Sep 2023

Issues with Channel filter bot
  • Resolved
    Resolved

    We have completed the cleanup and confirmed the bot is fully operational again. Thanks for your patience, this incident has been resolved.

  • Monitoring
    Monitoring

    We have developed and implemented a fix for the second issue. The bot is running again and we are in the process of cleaning up and ensuring the bot is functional again

  • Identified
    Update

    We have fixed the database error but encountered another error with data entity metadata not being available to our application. We are working on a fix for this

  • Identified
    Identified

    We have identified the issue to be caused by an automated database version upgrade. We are in the process of fixing the issues and checking if other services are affected as well

  • Investigating
    Investigating

    We are currently investigating reports by our monitoring service that our channel bot is experiencing issues.
    We have confirmed the bot to not be responding to interactions and are in the process to search for the root cause.

Webhosting experiencing downtime
  • Resolved
    Resolved

    We have identified the storage exceeding to be linked to our cloud service and a missing configuration to auto delete trashed files after a set amount of time. We updated the config and are monitoring monitoring this to be resolved.

    This incident has been resolved, all systems are operational again.

  • Identified
    Update

    We have restored access to all our services and identified the root cause of the downtime which was exceeding the storage quota on our webhosting. We are now investigating the cause of the storage exceeding and try to prevent this issue from happening again.

  • Identified
    Identified

    We are currently experiencing downtime on a few of our systems. The issue has been identified and we are working on a fix to resolve this. The fix will take approximately 90 minutes. Thanks for your patience and sorry for any inconvenience caused.

Sep 2023 to Nov 2023