Hey, there! Log in / Register
MBTA replacing the failing cabling system that was the reason the Red Line driver left what became the runaway train
By adamg on Wed, 03/02/2016 - 8:26am
A MassDOT report concludes the driver of the infamous train was to blame for it heading towards town by itself, but notes the T is also aiming to fix the condition that led him to leave the train in the first place - problems with an old cabling system leased from Verizon that was making signals fail and trains stop on the tracks south of North Quincy.
The Signals & Communications Department has appropriated funds to extend a signal trough from North Quincy to Braintree Station. This would replace the existing, aging data cables and provide a more robust signal without reliance on Verizon services.
The report, publicly released yesterday, says operator David Vazquez wrapped his cab's microphone cord around the trains controller, defeating the purpose of its "dead man's switch" so he could put on some gloves as his train left Braintree station shortly after 6 a.m. on Dec. 10.
He didn't get far: The train stopped just outside the station because of a malfunction in the Red Line signaling system in that region - a malfunction that had been deviling riders for a couple of days before Vazquez's train started galloping toward the Neponset River.
Red Line cars have a switch that lets trains continue through malfunctioning signals, at a maximum speed of 25 m.p.h., but it's located on the outside front of the cars. After getting permission from the Red Line operations control center (OCC), Vazquez jumped out to flip the switch - after forgetting to unwrap the microphone cord and forgetting to set the train hand brake.
In its report, MassDOT notes the signals had been failing because of a problem in the cables leased from Verizon in that stretch of the Red Line:
Prior to the incident, from December 8 through December 10, 2015, both the Normal and Standby MBTA equipment logged communication and signal failures. The failures were intermittent, lasting from 20 seconds to a couple minutes at a time.
On December 8, in response to the failures, maintainers tested and rebooted the MBTA Programmable Logic Controllers (PLCs) to troubleshoot the faults. After testing, MBTA equipment was found to be functional and the failures were attributed to a poor connection on the leased Verizon signal network.
The report continues that the T had to spend $500 to replace a door window a passenger was trying to kick out in an escape attempt as the train sat dead on the tracks - stopped by having the power on the track shut - just past North Quincy station. The first inspector who got on the train - using a southbound train commandeered from service:
Determined that passengers expressed concern as to the whereabouts of the operator, but did not seem particularly alarmed or distressed by the event. There was no indication of panic.
MassDOT also praised supervisors at the operations control commission for dealing with something they'd never dealt with before in a calm manner that prevented what could have turned into a catastrophe:
The investigation confirmed that the OCC and power dispatch teams under the direction of Mark G. McNeil, OCC Supervisor, effectively and safely brought the subject train consist to a stop, utilizing a series of actions to clear the track way of potentially conflicting trains, and power disruptions to eliminate propulsion. The processes used were based on best practical judgment, falling outside the scope of established rules, procedure, special orders or practices.
MBTA Safety’s analysis of unattended train movement incidents nationally and internationally determined that such unattended incidents resulted in either a derailment or collision with another train, equipment or stations. The consequences of such events entailed loss of life, injuries and damages. The actions of the OCC and power dispatch teams are not only commendable, but represent a highly effective risk crisis management process to address a volatile and transient event. The lessons learned benefits of the team’s effectiveness in handling this circumstance should be incorporated into future training and procedural improvements.
Complete MassDOT report, via WBUR.
Neighborhoods:
Topics:
Free tagging:
Ad:
Support Universal Hub
Help keep Universal Hub going. If you like what we're up to and want to help out, please consider a (completely non-deductible) contribution.
Comments
What about the signals?
Could we fix the signals? Or maybe the switches? Because by my count those go wrong about 30,000 times per day, while the problem they're now solving has manifested once.
(Mostly snarky here. Mostly.)
ahem!!
problems with an old cabling system leased from Verizon that was making[ signals fail ]and trains stop on the tracks south of North Quincy. Read more.
Refund
So they're getting a refund from Verizon for their lack of service right? Surely they included some type of uptime requirement in their SLA with Verizon for the lease.
time to consider changing all of red line to buses
Time to think about whether the red line is beyond repair, and should totally be replaced by buses, not just the Mattapan section. Commuter rail could carry more Braintree and Quincy residents from the stations it already stops at. If Baker cancels the red line cars being made by the Chinese company, he could use that money and some of the money for all the signal problems to pave the red line and order new, state of the art buses that will make other riders jealous. I was cynical when the state (think it was GM Prince) said the same thing about the silver bus (green line riders would be jealous we were told), which was substituted for a planned light rail. But they were right, silver line like service for all the lines is the future.
Horrible Idea
The Red Line runs at 4 minute intervals and is still packed to gills on trains that hold thousands. Ignoring the fact that they'd have to rebuild every station at the cost of billions there is no way busses could handle the red line capacity even on a dedicated roadway.
If anything else they'll need to the opposite and expand the platform length to accommodate longer trains if ridership keeps growing as projected.
The commuter rail can handle
The commuter rail can handle many of the people coming from Quincy and Braintree, they already pick up passengers at both those stattions and at JFK. And the commuter rail has a lot more capacity, ridership has declined as far as passenger growth.
http://pioneerinstitute.org/blog/mbta-commuter-rail-only-system-in-count...
This is win win, it decreases costs for running the subway while adding passengers to the underutilized commuter rail. The silver line is packed to the gills and passengers have to wait for several buses to pass before we can get on, why are red liners so special? Baker proposed it for the Mattapan line, why not the whole thing? And the green line was cancelled in somerville (the city in MA with highest pop density). Those people are stuck with buses, why not red liners?
I agree
Rip it all up, pave it over, go to fossil fueled busses. Old ones. Leftovers from other cities. Rails are outdated by a hundred years! Using electricity to power transit is a fool's errand. Why, next you'll be telling me the state/MBTA should use its rights of way and other land holdings to put up windmills and solar panels so the system could power itself! What a pipe dream...
Double the fares too! That'll bring in some money. Privatize late night services (done!) AND the Mattapan line. The poors will love it. FREE MARKET BABY!! Also instead of running their own infastructure they should continue to cotract and sub-contract everything to private companies and then pay them forever to maintain the sub-par work they did!
There is clearly no future in mass transit! People should just work within walking distance of their homes, or just drive.
Makes sense to me!
Um...
What about service to Quincy Adams, North Quincy, and Wollaston stations? All 3 stations have heavy am/pm commuter crowds, and none are served by the commuter rail. Quincy's streets can't support more buses, and there's nowhere to put a dedicated bus lane.
If people want the train they
If people want the train they can drive or bus to the ones that do offer commuter rail. If not, they can take the bus. Why are people in Quincy and Braintree so much more important than other places in metro Boston? Listen, Baker ran on no new gas tax, repealing scheduled increases like ones that the T riders get, and he's in charge. He is saying we cant afford the Mattapan line, green line, etc. So it seems logical to save some money on the Braintree line and have commuters there take the commuter rail or buses to Boston. There isn't unlimited money and theBaker has spoken.
It was GM Mulhern who was a
It was GM Mulhern who was a former bus driver, a big proponent of wiping out the Boylston Street tunnel, and now the notably crooked head of the "we ain't showing the books to anyone" MBTA pension fund.
Great idea, but...
Why bother wasting money to pave the red line right of way? Just pack those thousands of commuters of buses and send them right up the Southeast Expressway. Boom, problem solved.
I think I'm starting to get
I think I'm starting to get it, what with this guy's username and all. He's supposed to be some kind of over-the-top parody of an irrational anti-Republican ideologue, and not actually meant to be taken seriously. He's pretty good at it, too.
Money Money Money
I've managed leased Verizon lines. (AKA Dry line or PAS lines.) They aren't cheap. I was paying a few hundred a month per line and this is with a good corporate rate for a non-critical service. I'm sure the MBTA has hundreds (thousands?) of these circuits.
So how much has the MBTA paying for these? And for how many decades have they been paying them?
Maybe instead of closing the deficit by cutting service they could close the deficit by not paying for things they can do themselves, like running a [fucking] cable down their own right of way with their own crew
This is like owning a driveway yet paying top dollar to rent a spot in a private lot next door.
so have I
I used to run a private WAN all on leased lines from Verizon.
I said this before when this issue came up and I'll say it again. You have to understand a bit about telephony history before you can understand why the T leased lines from Verizon
If you go back prior to 1984 (prior to the Bell Divestiture), anything communication based, you often just went to the phone company for service. It was very, very rare that it was done in house. Even in office buildings, your phone system was usually Centrex lines, unless you were big enough to warrant your own ESS1 switch in your basement (like a few very large office buildings in DTX still have). But even then, the phone company still managed and maintained those lines even though 100% of the usage was for your office building or essentially one customer.
Same with the T. Unlike today where everyone and everyone's brother does tech, back in the 70s and 80s (when that section of the Red Line was built), it was not cost effective to run, maintain, and manage your communication lines. Since the T has a wide network of tunnels, and the phone company had the technical know how to run such a network, they worked together. Verizon got tunnel access to the lines and got the ability to sell products and services using those same lines to non-MBTA customers. The MBTA, out of the deal, got someone to 100% manage the infrastructure instead. No need to hire anyone with any technical know how about those lines because the phone company did all of that for you and already had the knowledgeable staff on the payroll.
And as I've said before, if it wasn't broken, don't fix it. As a tech person, I'm appalled that this is still used today since POTS lines are pretty much a dinosaur. However, as someone who used to support legacy systems (1970s-era POS systems), I can see why it was left alone. If it wasn't broken, why fix it.
And until about 10 years ago, it was still cost effective to allow the phone company to continue to manage these lines. Now that Verizon has let its copper network deteriorate, this is what happens. Nor does Verizon have any inclination to fix any of it (much like the same treatment they give residential and business customers).
Do I think the T should bring it in house? That's a hard question to answer, I would need to see staffing and payroll numbers to staff such positions in house at the T vs what they pay for the managed services from Verizon for these lines. And then the cost of actually paying for cabling and pulling new cables should be considered too. Yes train service is affected, but the T would have to weigh out the two options and probably seek out the lower cost one. (which IMHO would be to stay with Verizon)
The other problem is that.. there is no other provider out there who could or would want to take over such an arrangement. POTS is a dying breed, and to re-pull lines or switch stuff over would be a huge capex expense.. too much of an expense for any reasonable phone company to take over the contract.
So unless the T wants to string their own cable, hire their own workers on the payroll, we're pretty much stuck with Verizon. Yup It sucks, but welcome to monopolies.
Not sure I agree
Perhaps in the 80s what you said was true. But it's not as if the MBTA doesn't have a signal department and isn't accustomed to this sort of thing. I don't suggest any of this is "easy" but I'd be surprised if they didn't already have people and own their own lines elsewhere.
Regardless, every time there is costly, reoccurring payments to a private vendor it should be asked how much it would cost to do this sort of work in house and not pay the fees. If the MBTA could spend $1 million today to save paying $50k/month into the future than this sort of investment is reasonable even if it entails adding one or two people to the payroll.
kinda
You're confusing two very different things.
Management of Communication Networks
Management of Signal Networks
Yes, you can have in-house people who manage the signals but not the comm networks that intersect them. And I'm sure the T already has these people. So it's their job to manage from the signal itself to any wiring it needs to have done (control boxes, etc) TO the interface with the phone network. Then Verizon would take it from there.
Think of it this way.. like your home POTS line. You are required to wire, maintain, and manage the wiring in your home to the NID/Demarc on the side of your house. The phone company manages it from the NID to their central office.
You wouldn't build a home and have your wiring person wire up all the jacks in your home AND pull the wire from your house to the pole on the street. That's the phone company's job. The wiring person instead brings it to the demarcation point and the phone company takes it from there.
As someone who works for a company who outsources everything.. we've done the numbers, it is cheaper to out source our email to Microsoft than to host exchange servers in house. Its the same with the T. It seems like it would the reverse, but in reality, its actually cheaper in the long run to farm it out to someone else.
and 50k? Any good telephony person with a deep understand of all of this would cost over 120k in the private sector. That's just to MANAGE the network. And you'd probably need a team of 10-20 people just to manage this network. (Yes really, considering how vast their communication network is). So you're taking 1.2 Million just in salary just for 10 people for one year.
Wiring is a whole separate issue, and since the T is a union shop, you would have to use union electricians to do the work so that will run you about 150/hr or more. While it may seem the up front costs would eventually fizzle out.. but every time there's a problem, you have to hire one of those union guys again to fix it. And you'd have to absorb those costs vs having it included into your SLA.
So let's do some math here.. let's say it cost the T 20 Million to re-wire all its signals to their own in house communication network. And let's say it cost a 1 million a year to Verizon to manage such a network. It would take 20 YEARS to pay off the original investment. This cost does not include maintenance on the wires itself (if something breaks), or even the 'in house' staff to maintain everything. And in 20 years, that system may be so out of date that there's no one around to fix it so you may be forced to upgrade again, at your expense.
Verses if you paid Verizon, the maintenance, upgrades, and what not are included in your contract. So in the end when considering everything that it would entail, it's cheaper just to have the phone company still manage it. (And yes, I'm well aware that the argument there is that Verizon isn't doing their job, however that is more between Verizon and the T's Legal department for breach of contract for Verizon not living up to its end of the bargain).
I do get your point. but I've done some cost analysis for in house vs out source, and while I like to have staff I can manage in house, sometimes it is better and most cost effective to just outsource it. And considering the T's track record with maintenance, it seems like to keep such a critical system going without many issues, outsourcing it to be maintained better, might be a better option also
Edit: If you're reading this and keep refreshing to see new edits. Sorry, this is a complex issue with lots to talk about.
Numbers
You might be right or this might be a case of the T not spending money now to save money in the future. Obviously the signal network is huge. Is Verizon handling these lines for the Alewife branch? How about the Blue line and older networks such as the commuter rail? If Verzion only accounts for 20% of the total system signal distribution but represents 40% of annual spending on signals there could be a scam. (Particularly if their wires are more failure prone.)
Maybe the T has done the math and found it to be cost prohibitive. They never release this stuff to the public so it's just speculation on our part one way or another. But I would argue that a hypothetical 20 year investment is reasonable when considering the importance of the lines AND the fact we're talking about a system which is unlikely to see any major changes for the next 20 years.
The T is also known for being ripped off horribly by outside contractors (See: GLX) and Verizon is known for being scummy and expensive for governments so I'm going to take any claim of a need to use their services with a huge grain of salt.
This gets back to basic maintenance. The T would rather forgo basic upkeep even in the face of horrible outages in the future. The current fleet of subway rolling stock might be fine if it was rehabbed every 7-10 years at a fraction of what it cost to buy entirely new fleet every ~30 years.
Yes
I do openly admit that my hypothesis is entirely IMHO and from my 20+ years or so working with telephony stuff and contracts. So I could be wrong, but if the T followed industry standards set way back in the day, it should just about be the same. Because prior to the 1990s, you often just went to the LEC ("The Phone Company") to do your communication work.
I'm willing to be it was ALL of it is being managed by Verizon. Red, Orange, Blue, Green, and Commuter Rail. Even more so the commuter rail. Many rail lines have telephone (and now Fiber optic) cables buried beneath them because the right of way goes on for miles. (Its the easiest to build out since no demolition has to be done to put said wires in.). Even Grand Junction has a Fiber Optic cable that runs thru most of it and connects to the regional fiber optic network.
But also keep in mind that many rail companies (i.e. Amtrak) manage their own communication networks because those lines existed in the days of telegraph.. long before Ma Bell was ever around, so it remained the same. The MBTA may contract the management of those lines to whoever actually owns the tracks. Of course, this still leaves sections of the CR that are MassDOT/MBTA owned and operated. (which means it could very well be Verizon doing the management for those lines)
And going back to my Pre-1984 argument.. Any line except for the Alewife Branch was planned and built prior to 1984. And even sections of the Alewife extension were being built 1981-1988, so that is probably was broom'd under the Verizon contract. Again, in the 1980s it was probably cheaper just to amend the contract to add the new rail line communication networks. If they are already doing 80% of the communication network, it makes logical sense (at the time) to let the LEC take care of it also. Plus I'm sure the communication lines under that branch are fairly new and rely more on Mid-1980s technologies (early fiber, etc), rather than POTS copper.
Correct. However, if Baker wants to fix the signalling system like he says he does.. they will be forced to upgrade to a non-legacy system (i.e. IP based, etc). And who knows, in another 20 years, we may all be wireless and wires may be a thing of the past so it'll be time to upgrade again. (gotta love tech, it's always changing)
Yes I agree. In my workings with Verizon, they are a sleazy company to deal with and yes, they could very well be ripping the T off. But it's all in what is in their contracts. Remember, a contract with the MBTA isn't just the MBTA, but the entire state. And many of these contracts have existed for decades, and may not renew for decades at a time (they could be 10-20 year term contracts), so even if they are being ripped off, there's very little the T can do to re-negotiate the contract without some legal action.
I still do stand by my argument about seasoned talent. It's very hard to hire someone who is seasoned in legacy systems. As time goes on, these people are becoming less and less avaliable. Many are retiring and leaving the industry because the days of copper telephony stuff is becoming a thing of the past. And then of course cost.. an educated guess tells me that they can't just hire anyone for the job. They need someone with massive experience doing similar work, along with working with very antiquated equipment. Some of which may have not been used in mainstream business for many, many years. That comes with a very high cost associated with it. You want talent and experience to match. Be prepared to pay for it. Something I think the T does not have the money for.
This is where Verizon is the winner. They, as a LEC, have over 100 years experience in telephony. They also have a high retention rate on older employees. So it's clear they have the talent to manage such a network. And they also have way more leverage at getting such talent than the MBTA ever could.
I think you're coming around to what I was getting it when I said something above about the T's track record about maintenance. They just do not maintain anything very well at all. I cannot imagine what shambles the communication network would be if it was brought in house years ago.
Beating a dead horse
I think we're the only two people having this conversation and we mostly agree but...
No longer the case. Verizon has been laying off line workers for a while now or transferring (selling) them to outfits like Fairpoint who have even worse track records. Given the number of failures as of late it's hard to argue they are offering premium service. While Verizon might make some nice cash from these contracts the company as a whole isn't focusing on these sorts of things any longer.
On the other hand, the MBTA is accustomed to having dedicated people on staff to work on highly specialized and archaic systems. Verizon has an incentive to layoff an expensive, aging lineman in the goal of cutting costs in ways the MBTA won't.
It's hard to tell how many of the signal problems as of late are the result of Verizon directly but delayed service hurts the MBTA's customers a lot more then Verizon who at best will pay a small plenty for the downtime. Had the MBTA replaced Verizon's lines years (decades) ago there is a good chance it would have cost MBTA less in the long run and provided better service. After all, that's why they are installing their own lines now.
Getting back to your example of outsourcing an email server, that's a place where there is real competition and a low cost of switching providers. Contrast this to dedicated long-haul dry contacts where Verizon is the only option and thus they are going to charge monopoly prices. They have little incentive to offer high quality service particularly if the MBTA is too spineless to take them to court for breach of service contract. The MBTA would do well to analyze these sorts of contracts as cost savings.