WorldDialPoint
Please Login

Network Issues

Maintenance Required by software vendor (Resolved)

Affecting Other - All services | Priority - Critical

Maintenance Required by software vendor

Last month there was a software patch to our cloud instances which we thought was fixed, it's now been brought to our attention there needs to be another patch to ensure stability is maintained.

 

So now we are booked in for this Friday night 9PM to 11PM the same time, we finally have a solution patch from the software vendor which has been tested thoroughly in a replicated Dev cloud environment. 

Again it will be around 15 minutes from start to finish

We do apologize for any inconvenience caused

UPDATE 22:45 - Maintenance Finished. Total down time 10 mins, everything back online


Date - 15/12/2017 22:30 - 15/12/2017 22:45
Last Updated - 16/04/2018 19:16


Emergency Maintenance Evoice Stsrem (Resolved)

Affecting Server - Evoice VoIP Service | Priority - Critical

There will be some interuption on the evoice system to reselove an issue that should clear the issue we had last Friday 13/04 at 16:00

Engineer will be working on the cloud 27.50.78.2 for approx 30 mins between 21:00 and 22:00 and there could be some interuption with incoming and outgoing calls and should not last more than 10 minutes.

We appologize for this inconvenience caused. Date - 16/04/2018 21:00 - 23/04/2018 14:52
Last Updated - 16/04/2018 19:15


Packet Loss on our links (Resolved)

Affecting Server - Business Service | Priority - Critical

We are seeing random packet loss on our links and have reported it to the data center. We were advised that they are trying to mitigate the issue and clear all the random packet loss

More updates to come

UPDATE 16:29 - The data center is trying to advertise our IP range out of Vocus

UPDATE 12/03/2018 17:00 - Issue now fully resolved. The issue was randmly packet loss of an interval of approx 2 hours  for a duration of 2-3mins on AS 45671 via Vocus. Now AS 45671 has been routed via Equinix, you will only see vocus if your provider is peering with in brisbane but in saying that the packet loss is not happening anymore via vocus peers in the last 24hrs. Date - 08/03/2018 15:00 - 12/03/2018 17:00
Last Updated - 12/03/2018 17:07


Migrating of Prifixes to new switch (Resolved)

Affecting Server - MOR | Priority - Critical

Tonight we will proceed in moving our prefixes over to a new switch to where the voice cloud servers are, and from a core device we are looking at decommissioning in the future.

This will take the below prefixes offline for up to two minutes while they are dropped and picked back up in OSPF on the new switch:

27.50.78.8/29
202.130.35.64/29
27.50.78.0/29

Date - 05/12/2017 21:00 - 14/12/2017 15:06
Last Updated - 05/12/2017 16:15


Data Center Cloud Maintenance (Resolved)

Affecting Server - MOR | Priority - Critical

We just have been informed from the data center that a reboot is required due to some urgent maintenance on one of the cloud services.
This will be impacting only services that are conecting to IP 27.50.78.10 no other services will be interupted.
Services will be unavailable for approx 10- 15 mins between 22:00 - 23:59 AEST tonight

We could not get an exact time frame due to the complexity of the data center cloud services but expected to be at around 22:00 AEST

We appologise for the short notice but this has also got us by surprize too.

UPDATE - 20:17 Reboot of cloud services finished
. Date - 10/11/2017 22:00 - 05/12/2017 16:14
Last Updated - 10/11/2017 21:21


Phone numbers certain batches (Optus Issue) (Resolved)

Affecting Server - Business Service | Priority - Critical

On the 02/11/2017 we noticed that out main number 0731077420 was not receiving calls fromthe mobile and pstn network accross Australia, we then have put a fault with our carrier for Urgent fix.
After a while we started receiving email that some other numbers are also not working.

After extensive diagnostics with checking the numbers, we have found that there is an issue with some ranges of phone numbers that they cannot receive any calls from any network in Australia or overseas.
We have esaclated the issue to the appropiate carrier straight away

Phone numbers that are ofline are in the ranges as follows:
073107xxxx (back online at 7pm on the 02/11/2017)
029007xxxx (back online at 3pm on the 03/11/2017)
039912xxxx (back online at 1:30pm on the 03/11/2017)
088122xxxx (back online at 4pm on the 03/11/2017)

UPDATE 16:00 Optus has advised that all there numbers efected are back online

On a different note we will be moving any numbers away from Optus that we have left to our main carrier.

This is the whole incident report from Optus

Updates from upstream:

03-11-2017 12:06

We currently understand this issue to be affecting inbound calls for some ported numbers that are being delivered by Optus.
Optus have scripted a solution for this problem which is scheduled to finish running as of 1pm AEDT and will see numbers restoring as they are reconfigured on the network.

We have escalated this issue to our Account Manager but are unable to have our numbers prioritised due to the nature of the fix for the fault.

We will provide further updates as they become available.

 

03/11/2017 07:00
Internal team has successfully tested 25 number ranges after testing a re-configuration deployment. They have identified a total of 34,000 number ranges affected by the outage. Internal team has advised rollback action plan and engaged Tier 2/3 to support in deployment. They have begun rollback of select service changes with a tentative ETR of around 5 hours

03/11/2017 03:00
Switching resources have advised that too many services remain to be restored (around 28,000 number ranges) to be completed manually. We are now awaiting an update from our internal team regarding their plan to bulk change configuration to restore all services. No ETR is yet available but a further update is expected during early morning hours.

03/11/2017 01:00
Switching team working alongside Tier 2 to manually restore services is still ongoing. Tier 2 engaged internal team to see if changes can be scripted and deployed for faster resolution. No ETR yet available for full restoration

02/11/2017 23:00
Optus switching engineers continuing to restore services 100 number range at a time. Services will slowly be coming back over the next few hours but no ETR is yet available

02/11/2017 21:00
Optus Engineers have recognised a fix and are now manually correcting the ranges to have been affected. The root cause is still being investigated at this time. There is no ETR yet but services are slowly being restored.

02/11/2017 19:58
Optus has received several reports of calls failing to individual numbers and ranges that appear to have been ported to another network.

At this stage the root cause has not yet been identified however our Incident Management team have engaged the relevant fixer groups to investigate and restore services.

This case is of the highest priority within Optus Wholesale and Networks.

There is currently no ETR but we will provide further information as soon as it becomes available.

Date - 02/11/2017 14:04 - 03/11/2017 16:00
Last Updated - 10/11/2017 19:40


BBvoice Network Outage (Resolved)

Affecting Server - Business Service | Priority - Critical

BBVOICE network  (27.50.78.10) has lost a core router.
Enginners has been already despached to the data center at 7:am when monitoring started giving errors

This is effecting incoming calls customers who are conecting via 203.176.185.10 and all incoming calls

ETA at this moment is 08:45 am

Core router changed and service moved on to it.
Service fully restored at 8:00am

Issue restarted (it is not the core router) currently have engineers at our data centre rectifying the issue.

UPDATE - approx. 10:00am - 10:15am EST

UPDATE 10:21 - ETA changed we should receive the next update bu 10:35 am

UPDATE - 10:36 - ETA Moved 11:15 am

UPDATE 11:25 - Engineers are still working on the issue, we will update again within the next 30 mins

UPDATE 11:48 - We just received from the Data center in MLB that core Components of the network will be UP and running within the next hour and then external connectivity will follow

UPDATE 12:20 - SERVICES ALL ONLINE
Date - 13/09/2017 07:40 - 08/10/2017 06:43
Last Updated - 13/09/2017 15:17


Voice Outage (Resolved)

Affecting Server - Business Service | Priority - Critical

There is an outage in Servers Australia network that is not allowing traffic to flow properly and thus no audio or no connection.
At the moment there are engineers on site looking into it

UPDATE: 13:56 Network engineers have been alerted to a down transit port in our Vocus Doody St data centre.

UPDATE: 14:05 Transit links back online. Services also back normal

UPDATE 14:23
We have received confirmation
Engineers have restored connectivity to all offline services and our provider port is now back online.
Date - 21/08/2017 13:51 - 04/09/2017 07:29
Last Updated - 21/08/2017 14:24


One way Audio on random numbers (Resolved)

Affecting Server - Evoice VoIP Service | Priority - Critical

Details are as follows:

 

-------------------------------------------------------------------

Type: Un-Planned

Severity: Critical

Period From: 2017-08-11 08:30:00 AEST

Period To: 2017-08-11 00:00:00 AEST

Location: Melbourne Only

Services Affected: One Way Audio and no DTMF

Detail:

The issue with our upstream carrier has started again this morning.

Optus have been advised and we hope they will rectify the issue soon.

UPDATE - 11/08/2017 at 09:51 am

Optus have done a temporary fix by resetting their equipment to resolve the present issue.

Optus will be relocating all our numbers that is effected to a new switch during the course of the day for the permanent.

Our Carrier was already in the planning stage of moving our numbers to a new interconnect away from Optus that will provide far greater control and will allow us to be more nimble and to act much faster if any issues arise. Unfortunately when problems have occured on the Optus network they move slowly to resolve.

UPDATE - 11/08/2017 at 10:15 am

Issue have restarted again, Optus has already been notified, hopefully they will restar the equiment again.
Will keep you updated

UPDATE - 11/08/2017 at 10:45 am
It seems to be working again but at this time no update from the carrier as yet

Date - 11/08/2017 08:53 - 21/08/2017 13:53
Last Updated - 11/08/2017 10:47


Issue with Routing and IPV4 in data center (Resolved)

Affecting Server - Business Service | Priority - Critical

Engineers at the SAU data center are investigating an IPV4 issue affecting services in Sydney. Updates to follow.

Update 1 19:20 - Issue seems to be now resolved but we do not have any confirmation yet from the SAU Data center. We will update once we receive full confirmation

Uptae 2 20:04 - Data Center just confirmed that the issue is fully resolved
.
.
Date - 19/06/2017 18:29 - 11/08/2017 08:55
Last Updated - 19/06/2017 20:06



« Prev Page   Next Page »


View RSS Feed


Quick Navigation

Client Login

Email

Password

Remember Me

Search



Follow Us on Twitter

Copyright © WorldDialPoint 2002 - All Rights Reserved

Terms & Conditions | Privacy

Designed by LogoSpecialist.net