• Announcements

    • MannDude

      Current state of vpsBoard   02/04/2017

      Dear vpsBoard members and guests:

      Over the last year or two vpsBoard activity and traffic has dwindled. I have had a change of career and interests, and as such am no longer an active member of the web hosting industry.

      Due to time constraints and new interests I no longer wish to continue to maintain vpsBoard. The web site will remain only as an archive to preserve and showcase some of the great material, guides, and industry news that has been generated by members, some of which I remain in contact to this very day and now regard as personal friends.

      I want to thank all of our members who helped make vpsBoard the fastest growing industry forum. In it's prime it was an active and ripe source of activity, news, guides and just general off-topic banter and fun.

      I wish all members and guests the very best, whether it be with your business or your personal projects.

      -MannDude
drmike

OVH Canada Offline

25 posts in this topic

OVH is offline in Canada right now.

Someone miles away from the DC cut fiber somehow.

What is alarming here is that fiber goes to Newark, NJ, meaning it appears OVH only has that bandwidth.  No local / regional peering.  Quite strange.

6 strands are down.  OVH CEO said 4 hours.  

It's dark fiber, no clue how big the whole bundle they are in - might be a ton of fibers to cut, prep, polish, test, etc. before they get things patched up.

Share this post


Link to post
Share on other sites

Little fact correction on strands down... 26.

6x100G
20x10G

1 person likes this

Share this post


Link to post
Share on other sites

Congrats to Oles for entrepreneur of the year

 

Share this post


Link to post
Share on other sites

It appears OVH was doing a network upgrade also...  new bigger fiber was about to be brought online soon.   Think that's with the 100G lines were about.

They've gone quiet on updates.

Someone implicated a nearby bridge project as maybe being the source of the outage.   The bridge has a slew of demo and other stuff going on.. Zoo like any infra project is.  If fiber was cut on that bridge, good luck, working over water, might need to work suspended off the deck or boom arm from below.  

I am blah about fiber hauls over bridges..  Very common, but much crumbling ancient infra that gets pounded.  If bridge, someones ass is grass today.   Cause those are documented runs for certain and usually really visible to workers (aren't buried in soil).

No way this is 4 hour out.  They've been down / blips for 3 hours.  They will probably be back overnight to normal.  Sooner is some bad ass workers.

Share this post


Link to post
Share on other sites

 

Comment by OVH - Monday, 02 November 2015, 21:30PM

We still don't have an ETA regarding the repairs on the existing fiber.

However, it's been confirmed that 2 out the 3 new pairs of the Eastern route (which were being setup) are now usable. We are working with our providers in order to get a full connection as quickly as possible: the only cross-connect linking OVH to our provider (in Montreal) is currently missing.

 

 

1 person likes this

Share this post


Link to post
Share on other sites

Smells like they got caught with a scheduled event on the bridge or related construction plus lighting new fiber... 

Sounds like from that update, they are saying hell with the bridge (assumed) cut.  

Packets are still getting to OVH, but like 40-50% loss.

Share this post


Link to post
Share on other sites

Looks like my websites are finally back up!

Share this post


Link to post
Share on other sites

Issue with their network is far from resolved I'd guess.  Incidents are still open on that portal.

Couple of spots I checked show packets traveling to OVH now through TorIX (Toronto Internet Exchange).. Strange routes... I expect OVH to have fiber lit to TorIX, I surely would, directly:

 6  eth1-2.edge1.tor1.ca.as5580.net (78.152.34.199)  85.899 ms  67.502 ms  84.160 ms
 7  ovh.ip4.torontointernetxchange.net (206.108.34.189)  70.690 ms  72.053 ms  72.130 ms
 8  nwk-5-a9.nj.us (178.32.135.234)  84.419 ms  84.979 ms  85.306 ms
 9  bhs-g2-a9.qc.ca (192.99.146.101)  93.669 ms  93.327 ms  96.771 ms
10  bhs-3a-a9.qc.ca (198.27.73.94)  98.484 ms  98.150 ms  98.766 ms
 

So that goes into Toronto, into TorIX, then OVH rides it to Newark and all the way up to BHS.   Strange frickin design....

 

Share this post


Link to post
Share on other sites

Another OVH Failure , My friends are suffering and posting about it on FB lol

 

Share this post


Link to post
Share on other sites

That is lovely, I am truly suprised they do not have more outages. I have not had any issues with the service in their canadian facility I currently have a server in france and have no problems, although am looking to upgrade as it sometimes can lag.

Share this post


Link to post
Share on other sites

You'd think such a large provider, no matter how budget-oriented, would have a higher level of redundancy on primary fiber runs.

I subscribe to the school of thought that a DC with over 800Gbps of capacity and hundreds of thousands of servers shouldn't be brought down by something like this. Diverse fiber paths are fairly important when you're promising a 99.95% network uptime SLA.

Share this post


Link to post
Share on other sites

Immense packet loss to the point of unusability for a few hours yesterday. Could not reach ca.ovh.com from certain connections while able to load it slowly (excruciatingly so) from others.

According to StatusCake, no downtime at all for any of my servers there except a single brief moment lasting '00:07:07'.

1 person likes this

Share this post


Link to post
Share on other sites
3 hours ago, OSTKCabal said:

You'd think such a large provider, no matter how budget-oriented, would have a higher level of redundancy on primary fiber runs.

I subscribe to the school of thought that a DC with over 800Gbps of capacity and hundreds of thousands of servers shouldn't be brought down by something like this. Diverse fiber paths are fairly important when you're promising a 99.95% network uptime SLA.

I feel the same way about the fiber path issue...  Really unclear how much was and wasn't down.   I saw uptime on something there, but the packet loss and latency was so high that it was unusable.  Technically online, but real world it wasn't.

With OVH's SLA in place, expect claims and nose bleeds.  I await the Act of God exception to claims.  Can't see them going out of pocket for this blooper.

Share this post


Link to post
Share on other sites

They claim it's resolved now (ticket is closed) and they are working on having more fiber routes out of BHS.  They had something like that happen before and they did honor SLA claims.  I've been resisting temptation all day to buy one of their SSD VPS since I don't need it.

Share this post


Link to post
Share on other sites
1 hour ago, willie said:

They claim it's resolved now (ticket is closed) and they are working on having more fiber routes out of BHS.  They had something like that happen before and they did honor SLA claims.  I've been resisting temptation all day to buy one of their SSD VPS since I don't need it.

Unsure about matter closed.. Still showing open over here:http://status.ovh.com/?do=details&id=11304

Technically, it says In Progress, same as it has since it was opened :)

Routing has changed since earlier (mind you this is testing from Toronto proper):

 3  ovh.ip4.torontointernetxchange.net (206.108.34.189)  77.229 ms  77.542 ms  77.872 ms
 4  bhs-g2-a9.qc.ca (178.32.135.71)  78.173 ms  78.809 ms  79.113 ms
 5  bhs-3a-a9.qc.ca (198.27.73.94)  79.425 ms  79.745 ms  80.505 ms

Awaiting TorIX to catch fire now :)   Can see a lot of traffic better routing to Toronto as opposed to Newark.

This is from Chicago:

 5  * * eth1-7.core1.chi1.us.as5580.net (78.152.45.135)  62.422 ms
 6  eth1-2.edge1.tor1.ca.as5580.net (78.152.34.199)  85.348 ms  76.062 ms  75.424 ms
 7  ovh.ip4.torontointernetxchange.net (206.108.34.189)  223.170 ms  223.166 ms  223.198 ms
 8  bhs-g2-a9.qc.ca (178.32.135.71)  83.890 ms  86.239 ms  86.600 ms
 9  bhs-3a-a9.qc.ca (198.27.73.94)  86.963 ms  94.852 ms  95.160 ms

If lots of Chicago traffic gets siphoned via Toronto, could be an uptick in mass at TorIX.  Pretty decent BW growth there since August:

torix-year.png

Share this post


Link to post
Share on other sites

It's not like this is the first instance... a couple months ago, a car struck a telephone pole outside the DC and brought the entire network down.

1 person likes this

Share this post


Link to post
Share on other sites
59 minutes ago, OSTKCabal said:

It's not like this is the first instance... a couple months ago, a car struck a telephone pole outside the DC and brought the entire network down.

Bahaha? Really, this happened?  Link if you have one...

Share this post


Link to post
Share on other sites

  • Similar Content

    • By DomainBop
      OVH offers users  the choice of using its modified custom Linux kernels or the regular distro's kernel.  Ubuntu's license contains a clause that requires anyone who distributes a modified version of Ubuntu to get approval from Ubuntu and pay a license fee to use the Ubuntu mark. Canonical/Ubuntu wants OVH to pay a licensing fee of 1-2 euros monthly for each installation which over the course of a year would amount to about 1 million euros.  According to the article Canonical (owner of Ubuntu) is also asking Dreamhost, which also uses a modified version of Ubuntu,  to pay a licensing fee.
      (google translate was invented so you could read the French articles I link to) http://www.clubic.com/pro/actualite-e-business/actualite-809784-ubuntu-canonical-payer-ovh.html
    • By fm7
      Redirecting over and over to the login page.
       
      http://travaux.ovh.net/?do=details&id=17229
    • By DomainBop
      If any of you OVH based providers want to set yourself apart from the crowd and be the first to start offering PowerKVM VPS's out of RBX for under $7 monthly I might be interested.  A Power8 OpenStack offering would be even better as long as you keep it under $7  (it's been impossible to find a Power8 VPS since OVH/Runabove ended their Power8 lab).  :P
      ***
      SoYouStart launched a new server line with amazing specs for the price.:
      Virtualization options for Power8 servers: https://www.ibm.com/support/knowledgecenter/linuxonibm/liaam/liaamvirtoptions.htm
      MongoDB and MariaDB already offer Power8 versions so these servers would be great for heavy data processing with large databases...big data
       
       
    • By drmike
      Vancouver, BC December 14, 2015
      Canadian Web Hosting Acquires eSecureData, Further Expanding Datacenter Business in Canada
      Canadian Web Hosting, Canada's leading web hosting and infrastructure company, announced today that it has acquired datacenter and dedicated server provider eSecureData.
      eSecureData is based in Vancouver, BC and targets Canadian businesses and IT professionals who are looking for premium datacenter colocation facilities, Canadian dedicated servers, and Canadian Cloud Hosting.
      eSecureData is a long standing Canadian Infrastructure as a Service provider that has been delivering managed hosting, dedicated servers, colocation and domains to thousands of customers across Canada since 2007. The company is based in Coquitlam, BC and will be merging with Canadian's Web Hosting's industry-best staff and expanding its already impressive datacenter operations. An additional second eSecureData site is in the works in Toronto and will be added to the business in the next quarter. These datacenters will integrate with Canadian Web Hosting's cloud hosting services and dedicated CloudStream Network Backbone running across Canada and expand their presence to five datacenters in Canada.
      With this acquisition, Canadian Web Hosting will be upgrading its portfolio of hosting services and increasing its offerings catered towards businesses and corporations who require features like enterprise-grade infrastructure, SSAE 16 compliance and managed support expertise to help meet their requirements.
      Equally importantly, customers will see a significant improvement in performance and latency as Canadian Web Hosting expands its cloud initiative that is focused on improving capacity and performance across Canada as more of its cloud services come online. These include Canadian cloud computing, IoT and other services that requires faster and more software friendly infrastructure than what the majority of current legacy datacenter providers can deliver.
      "This is a huge opportunity to develop and grow our business and work with a world-class team. Canada is starving for leadership in the datacenter and cloud space and with services like AURO and Canadian Cloud Hosting, and their understanding of what comes next in cloud, Canadian Web Hosting is definitely that company," said CEO of eSecureData, Kevin Liang.
      "We are excited to be part of the team and to be backed by one of the leading Canadian infrastructure providers delivering managed hosting, colocation and cloud hosting. Canadian Web Hosting is a company that brings regulatory and financial strength that comes with their 17 years of leadership."
      Canadian Web Hosting has completed several acquisitions since 2014 and is continuing to build out its end-to-end Canadian infrastructure to meet the significant demand they are seeing. For more information about Canadian Web Hosting and eSecureData, contact [email protected]