WorldDialPoint
Please Login

Network Issues

Migration to new Hardware (Stage 1 of 2) (Resolved)

Affecting Server - Evoice VoIP Service | Priority - Critical

Stage 1 of 2 Migrating to High Availability and DRaaS Solution between 2 data centres (Equinix 1 and Equinix 2)

 

At midnight the 15/08/2018 Evoice SIP Softswitch will be migrating to a new hardware with extra redundancy between EQUINIX 1 and EQUINIX 2 in case of hardware failure in one data centre.

There will be some downtime involved of approx. 30 minutes.

 

In the last 2 weeks we have managed and prepared the new platform with continuous synchronize of the old platform with the major bulk of the data already moved.

The last step is to move IP addresses and sync.

Customers does not need to do anything as everything should come back normal after the downtime.

Stage 2 will involve the same for our wholesale system but this is planned for one week after.

UPDATE - 00:05 Migaration started

Update - 01:12 am Migarting Finished and all services online as normal



STAGE 2 

At midnight the 22/08/2018  at 00:01 am our Wholesale / Main SIP Softswitch will be migrating to a new hardware with extra redundancy between EQUINIX 1 and EQUINIX 2 in case of hardware failure in one data centre.

There will be some downtime involved of approx. 30 - 60 minutes.


UPDATE 21/08/2018 21:40 Stage 2 Migration moved to 23/08/2018 at 00:001am

UPDATE 23/08/2018 00:05 Migration Started

UPDATE 23/08/2018 00:35 am Migarting Finished and all services online as normal
.

Date - 15/08/2018 00:01 - 27/08/2018 18:10
Last Updated - 23/08/2018 00:57


Outage in data center (Resolved)

Affecting Server - Business Service | Priority - Critical

Current Status: Service Disruption

Started: 13/7/2018 9:39am (+1000)

Resolved:

 

Affected Infrastructure:

Components:  Cloud Services, Colocation, Dedicated Servers, Infrastructure, Managed Wordpress Hosting, Network, Service Desk, xDSL/EFM

Locations:  Brisbane, Brisbane - NextDC B1, Brisbane - NextDC B2, Brisbane - Syncom BNE, Melbourne - Equinix ME1, Melbourne - NextDC M1, Melbourne - Vocus, New Zealand, Perth - Vocus, Sydney, Sydney - Equinix SY1, Sydney - Equinix SY3, Sydney - Equinix SY4, Sydney - Global Switch, Sydney - NextDC S1, Sydney - Syncom SYD1, Sydney - Syncom SYD2, Sydney - Vocus

 

Update: Network engineers are currently investigating network interruptions.

Further updates will be provided when possible.

Status Page: https://status.mysau.com.au


UPDATE 10:08

Update: Engineers have confirmed connectivity has been re-established network-wide.

A PIR for this incident will be provided ASAP.

Please contact our helpdesk if you experience any further issues with your services.

Status Page: https://status.mysau.com.au


UPDATE 10:12

Update: Some services are still affected by this and currently being worked on.

Additional updates to come.


UPDATE 10:42

Engineers are still working on restoring services.

Another update will be provided shortly.

UPDATE 13:57

Issue has been resloved from SAU at 11:39am.
We have been monitoring since then and no issues reported

UPDATE 15:57

We just received a NOTIFICTION from SAU that they need to Reboot certain services at 16:30
This will take approx 3 mins to complete.
After this reboot  we are told that any other issues will be resolved.

UPDATE 16:10

Update: Engineers will be implementing a fix to resolve packet loss and latency issues at 4:30 PM AEST.

Between 4:30 PM and 4:40 PM clients may experience up to 3 minutes of downtime to services while engineers apply a fix to some devices in the Sydney, Melbourne and Perth network.

Date - 13/07/2018 09:43 - 17/07/2018 12:58
Last Updated - 13/07/2018 18:09


Outbound Call Issues on one of our carriers (Resolved)

Affecting Server - Evoice VoIP Service | Priority - Critical

Outbound Call Issues on one of our carriers started at 12:01pm

We have found that one of our outbound carriers was not terminating any calls or they had huge delay to conect and they were not sending any error back to us but instaed they were sending us that the call is being progressed and thus our second and third carrier could not kick in.

We have now eliminated this one carrier out of the mix at 12:40pm and now all outbound calls are conecting fine and no delay.

We also have requested this specific carrier for a please explain and what will be done moving forward so this issue will not happen again.
If we are not satified with the outcome then this carrier will be dropped permanetly.

We appologise for this mishap
. Date - 17/05/2018 12:01 - 13/07/2018 09:44
Last Updated - 17/05/2018 13:00


Emergency Vlan setting change required (Resolved)

Affecting System - 27.50.78.1 / 2 / 9 / 10 | Priority - Critical

Start time - 23/04/2018 22:00
Finish - 23/04/2018 22:30

We would like to advice everyone that the data center where all our cloud services are, needs to do some EMERGENCY work on our VLANS.

Due to this change there will be an outage of approx 30 to 60 seconds.
While this is happening no calls will be able to be made or received,

This is required, as one of the seniour enginners at SAU have found on all IP's on a particular ingress point to the network are not being correctly leaked out of the peering VRF into the switching fabric. This basically means that the IP's are taking an older, slower path that's showing packet-loss under a certain condition.
.

UPDATE 22:09 - Starting the changes now
.

Update - Work finished at 22:12 Date - 23/04/2018 22:00 - 17/05/2018 12:51
Last Updated - 24/04/2018 10:32


Maintenance Required by software vendor (Resolved)

Affecting Other - All services | Priority - Critical

Maintenance Required by software vendor

Last month there was a software patch to our cloud instances which we thought was fixed, it's now been brought to our attention there needs to be another patch to ensure stability is maintained.

 

So now we are booked in for this Friday night 9PM to 11PM the same time, we finally have a solution patch from the software vendor which has been tested thoroughly in a replicated Dev cloud environment. 

Again it will be around 15 minutes from start to finish

We do apologize for any inconvenience caused

UPDATE 22:45 - Maintenance Finished. Total down time 10 mins, everything back online


Date - 15/12/2017 22:30 - 15/12/2017 22:45
Last Updated - 16/04/2018 19:16


Emergency Maintenance Evoice Stsrem (Resolved)

Affecting Server - Evoice VoIP Service | Priority - Critical

There will be some interuption on the evoice system to reselove an issue that should clear the issue we had last Friday 13/04 at 16:00

Engineer will be working on the cloud 27.50.78.2 for approx 30 mins between 21:00 and 22:00 and there could be some interuption with incoming and outgoing calls and should not last more than 10 minutes.

We appologize for this inconvenience caused. Date - 16/04/2018 21:00 - 23/04/2018 14:52
Last Updated - 16/04/2018 19:15


Packet Loss on our links (Resolved)

Affecting Server - Business Service | Priority - Critical

We are seeing random packet loss on our links and have reported it to the data center. We were advised that they are trying to mitigate the issue and clear all the random packet loss

More updates to come

UPDATE 16:29 - The data center is trying to advertise our IP range out of Vocus

UPDATE 12/03/2018 17:00 - Issue now fully resolved. The issue was randmly packet loss of an interval of approx 2 hours  for a duration of 2-3mins on AS 45671 via Vocus. Now AS 45671 has been routed via Equinix, you will only see vocus if your provider is peering with in brisbane but in saying that the packet loss is not happening anymore via vocus peers in the last 24hrs. Date - 08/03/2018 15:00 - 12/03/2018 17:00
Last Updated - 12/03/2018 17:07


Migrating of Prifixes to new switch (Resolved)

Affecting Server - MOR | Priority - Critical

Tonight we will proceed in moving our prefixes over to a new switch to where the voice cloud servers are, and from a core device we are looking at decommissioning in the future.

This will take the below prefixes offline for up to two minutes while they are dropped and picked back up in OSPF on the new switch:

27.50.78.8/29
202.130.35.64/29
27.50.78.0/29

Date - 05/12/2017 21:00 - 14/12/2017 15:06
Last Updated - 05/12/2017 16:15


Data Center Cloud Maintenance (Resolved)

Affecting Server - MOR | Priority - Critical

We just have been informed from the data center that a reboot is required due to some urgent maintenance on one of the cloud services.
This will be impacting only services that are conecting to IP 27.50.78.10 no other services will be interupted.
Services will be unavailable for approx 10- 15 mins between 22:00 - 23:59 AEST tonight

We could not get an exact time frame due to the complexity of the data center cloud services but expected to be at around 22:00 AEST

We appologise for the short notice but this has also got us by surprize too.

UPDATE - 20:17 Reboot of cloud services finished
. Date - 10/11/2017 22:00 - 05/12/2017 16:14
Last Updated - 10/11/2017 21:21


Phone numbers certain batches (Optus Issue) (Resolved)

Affecting Server - Business Service | Priority - Critical

On the 02/11/2017 we noticed that out main number 0731077420 was not receiving calls fromthe mobile and pstn network accross Australia, we then have put a fault with our carrier for Urgent fix.
After a while we started receiving email that some other numbers are also not working.

After extensive diagnostics with checking the numbers, we have found that there is an issue with some ranges of phone numbers that they cannot receive any calls from any network in Australia or overseas.
We have esaclated the issue to the appropiate carrier straight away

Phone numbers that are ofline are in the ranges as follows:
073107xxxx (back online at 7pm on the 02/11/2017)
029007xxxx (back online at 3pm on the 03/11/2017)
039912xxxx (back online at 1:30pm on the 03/11/2017)
088122xxxx (back online at 4pm on the 03/11/2017)

UPDATE 16:00 Optus has advised that all there numbers efected are back online

On a different note we will be moving any numbers away from Optus that we have left to our main carrier.

This is the whole incident report from Optus

Updates from upstream:

03-11-2017 12:06

We currently understand this issue to be affecting inbound calls for some ported numbers that are being delivered by Optus.
Optus have scripted a solution for this problem which is scheduled to finish running as of 1pm AEDT and will see numbers restoring as they are reconfigured on the network.

We have escalated this issue to our Account Manager but are unable to have our numbers prioritised due to the nature of the fix for the fault.

We will provide further updates as they become available.

 

03/11/2017 07:00
Internal team has successfully tested 25 number ranges after testing a re-configuration deployment. They have identified a total of 34,000 number ranges affected by the outage. Internal team has advised rollback action plan and engaged Tier 2/3 to support in deployment. They have begun rollback of select service changes with a tentative ETR of around 5 hours

03/11/2017 03:00
Switching resources have advised that too many services remain to be restored (around 28,000 number ranges) to be completed manually. We are now awaiting an update from our internal team regarding their plan to bulk change configuration to restore all services. No ETR is yet available but a further update is expected during early morning hours.

03/11/2017 01:00
Switching team working alongside Tier 2 to manually restore services is still ongoing. Tier 2 engaged internal team to see if changes can be scripted and deployed for faster resolution. No ETR yet available for full restoration

02/11/2017 23:00
Optus switching engineers continuing to restore services 100 number range at a time. Services will slowly be coming back over the next few hours but no ETR is yet available

02/11/2017 21:00
Optus Engineers have recognised a fix and are now manually correcting the ranges to have been affected. The root cause is still being investigated at this time. There is no ETR yet but services are slowly being restored.

02/11/2017 19:58
Optus has received several reports of calls failing to individual numbers and ranges that appear to have been ported to another network.

At this stage the root cause has not yet been identified however our Incident Management team have engaged the relevant fixer groups to investigate and restore services.

This case is of the highest priority within Optus Wholesale and Networks.

There is currently no ETR but we will provide further information as soon as it becomes available.

Date - 02/11/2017 14:04 - 03/11/2017 16:00
Last Updated - 10/11/2017 19:40



« Prev Page   Next Page »


View RSS Feed


Quick Navigation

Client Login

Email

Password

Remember Me

Search



Follow Us on Twitter

Copyright © WorldDialPoint 2002 - All Rights Reserved

Terms & Conditions | Privacy

Designed by LogoSpecialist.net