Y Safle

I sent the following letter to the BT UK Correspondence Centre 3 times. I never once got an answer. I cancelled my BT subscription, but to re-use my efforts, I offer this sorry tale for your amusement.

Dear Sir/Madam

Re. Fault with BT SMTP Service Affecting Our Account

On or about Thursday 26th January 2006, BT Internet reconfigured their SMTP server service manually or automatically in some manner that caused a loss of service from our BT Internet account (xxxxxxxx.x.xxxxxxx@btinternet.com). We use a variety of laptops, desktop PCs and PDAs to access our account, depending on where we are in the house and what we are doing, and the failure affected all of these devices.

The symptoms of the failure were we could no longer send mail, because the SMTP authentication failed. However, we had not reconfigured SMTP authentication, and the password was clearly fine as we could continue to receive mail from the POP3 service, and use web mail.

Online Support

After waiting for a day or two to see if the problem would correct itself, my wife decided to contact technical support on Sunday 29th January, as she had an email she needed to send. She spent a couple of hours with the technical support to no avail. In this time, they made some suggestions that I think, as an ICT professional, were unacceptable, and you will want to review these issues:

1. At no point did the call centre take on board that this was a fault that had simultaneously affected multiple setups on diverse operating systems. When my wife asked if anything had changed on the mail server, she was adamantly told “no”, despite clear evidence to the contrary, and the fact that the call centre were clearly not in a position to know this.
2. On discovering that my wife was using Microsoft Outlook to read email, she was told that as this was not a supported mailer, she should take the issue up with Microsoft! Now this response annoyed me because
   1. the problem was very clearly not with the mail software,
   2. my wife could easily switch to Outlook Express if that is what was necessary,
3. If you say that you only support Outlook Express, and will not help with problems on any other mailer, then you lock yourself into proprietary Microsoft technologies. You are saying that users of other operating systems, handheld devices, text only mailers and mailers used by people with disabilities cannot access technical support. This is probably in breach of your duties under the disabilities discrimination act, and is certainly bad business sense, as you are saying that there is a huge user base that you do not want as customers. You do not seem to want users of open source software, for instance - despite the fact that your SMTP service is itself the open source qmail SMTP server.
4. Your support staff did not tell my wife what she needed to tell Microsoft. Certainly Microsoft have no interest in a sudden loss of service to the BT Internet mail service, and would send the problem straight back to you. This kind of passing-the-buck is not acceptable from a service that should be attempting to help customers resolve problems.

My wife, on my instruction, transferred to Outlook Express and refused to close the support call. After a while, the call centre asked if we had a firewall enabled. Quite sensibly we do run a firewall in our ADSL router, and the support centre suggested we disable it. I immediately disabled the firewall, and in the meantime established the further information that sending mail still worked from our dial up BT Internet account. However the call centre staff would not take our assurances that the firewall was disabled for granted, and informed us that we must contact our firewall vendor!!! Repeated insistence that the firewall was disabled eventually prompted the call centre to give us a telephone number through which we could escalate the fault.

Telephone Support

I called the support centre on Sunday evening to escalate the fault, and despite my telling the operator that we had been through online support, I was asked all the same questions. Yes, I had (grudgingly) booted my Linux laptop into windows and was running Outlook Express. No, we did not have a firewall running, and so on. At this point I was asked what ADSL modem we were using. I indicated that we had an ADSL router, and your call centre staff told me that the router must have a firewall and I needed to contact my vendor!

At this point, please bear in mind that (a) I had disabled the firewall, (b) the router had been working fine to date, and (c) the problem was failure of authentication, not failure to connect to the SMTP server. Again, this is a pass-the-buck attitude to support that must be dealt with. It is quite unacceptable.

I told your operator firmly (but quietly and politely) that the problem was with the BT SMTP server, and that he needed to escalate the fault - that the fault lay in a failure by the SMTP server to accept our fully resolved broadband IP address as a valid relay domain for our credentials. I reiterated that we had been through the online support process, and that the support centre had already been unable to help, and that the problem should be escalated to a suitable engineer. The operator refused to escalate at this point (some 10 minutes into the call) and continued to offer suggestions to fix the problem. I politely continued working with your staff to try various suggestions.

At one point your operator asked me to connect by hand to the SMTP service. This I did (I told him the port numbers as he started mentioning port 110, which is the POP service. Port 25 is SMTP). We quickly established that my communication was fine and I gave him my ADSL IP address. After many suggestions and many long periods of holding, your operator eventually agreed to escalate the fault some 50 minutes after I had place my call.

It seems to me that all companies I deal with who use outsourced support these days require me to speak to them for about an hour before they will eventually admit to pass on a fault to a technical contact. I am unhappy about this, and you really need to consider a means by which technical people can send technical queries to their peers, without this timewasting filter.

I indicated I would be going to bed at this point, and could the technical support engineer call me in the morning. The operator said he would pass on the message. No call was ever returned.

I would like a call credit for the time spent on this call - particularly as your operator refused to escalate the fault, and also because no call back was ever received.

Email Support

On the afternoon of Tuesday 31st January I used the BT web site to send an email about this fault. Prior to sending this email, on Monday night I spent some time investigating the fault, and discovered the following:

I connected by broadband and captured this SMTP session information: (I have obfuscated the base64 encoded password, but otherwise the session is exactly as seen on the SMTP server):

220 smtp810.mail.ukl.yahoo.com ESMTP EHLO eeyore.gloomyplace.org.uk 250-smtp810.mail.ukl.yahoo.com 250-AUTH LOGIN PLAIN XYMCOOKIE 250-PIPELINING 250 8BITMIMEAUTH LOGIN 334 VXNlcm5hbWU6 c3RsalcGdshabi2wLmtJJpzdG9u 334 UGFzc3dvcmQ6 aXblRmsqXQQ= 535 authorization failed (#5.7.0)

The WAN configuration for this connection was:

ppp0 Link encap:Point-Point Protocol inet addr:86.145.210.33 P-t-P:217.32.86.146 Mask:255.255.255.255

According to dig, the IP address 86.145.210.33 resolves as:

22.210.145.86.in-addr.arpa. PTR IN 604800 host86-145-210-22.range86-145.btcentralplus.com.

I then repeated the experiment with the broadband disabled, and connected via dialup. Here is the chat, which you will note succeeds:

220 smtp810.mail.ukl.yahoo.com ESMTP ehlo eeyore.gloomyplace.org.uk 250-smtp810.mail.ukl.yahoo.com 250-AUTH LOGIN PLAIN XYMCOOKIE 250-PIPELINING 250 8BITMIME AUTH LOGIN 334 VXNlcm5hbWU6 c3RlcGsaqhlfdfbdi5wsqasLdsd672 334 UGFzc3dvcmQ6 aXblRdsjkXaQ= 235 ok, go ahead (#2.0.0) ...

The dialup IP/gateway info was:

213.122.53.24/32 gw 213.122.53.24

This IP address resolves to:

24.53.122.213.in-addr.arpa. PTR IN 604800 host213-122-53-24.in-addr.btopenworld.com.

I sent all of this information with various suggestions in the email. I noted that the problem could be in the way that qmail is building its authsenders file (or PAM equivalent), and pointed out that the problem was that the btcentralplus.com. domain was not being listed against my credentials as a valid relay domain.

I then gave you the key information that would have allowed an engineer to resolve this problem. I wrote:

” It may be that this is because my BT Internet email address pre-existed the broadband connection”.

Other than an automated acknowledgement, no reply was ever received to this email.

Eventual Resolution

As BT were not talking to me, I continued my investigations. I noted that there was no huge outcry on Usenet about the failure of BT’s SMTP service, so I presumed that this problem only affected me or a small user group. I considered what was special about my BT account.

One thing that was special was that we have had a BT dialup account for a very long time. Long enough that we retained the five free email address service from our BT anytime account. Recently BT have extended this service to all customers, but it was a service we already had.

However it occurred to me that the automated registration system may treat my account differently from newer accounts with this service, and that you may have reconfigured your SMTP service to disallow the older accounts, or maybe my credentials were simply damaged in your database and needed refreshing.

I thus “upgraded” my account from the existing “five free email address” service to the exact same service! This will have pushed new versions of my credentials around the BT Internet network, and sure enough after a short pause, the SMTP service started authenticating from our broadband IP address once again.

Conclusion

It is now Friday 3rd February. BT have not followed up on my original fault report, nor on my email query. I have managed to resolve my own problem, but only through an in depth knowledge of how these services work. If we had followed your advice, we would now be talking with Microsoft!

I am extremely unhappy with this experience, and have noted various lessons that I believe BT must learn.

As a BT shareholder I am unhappy that you appear to be turning away a large part of your potential user base through your “outlook express only” policy. I am also unhappy that your failure to address such support issues is presumably turning away other customers. I myself am now considering looking for an alternative ISP.

I would like the following undertakings from BT:

1. To refund my call costs for my support call on Sunday Evening;
2. pass on my resolution of your fault to your call centre as soon as possible, so that others affected by this fault can be told quickly how to resolve it;
3. To address the failures in your technical support centre, to ensure that in future, genuine technical issues can be passed onto someone more knowledgeable more quickly. I would like BT to tell me how they intend to achieve this;
4. To ensure that the failure to return calls and emails is addressed;
5. To provide support for cross platform mailers - e.g. Mozilla Thunderbird, and also for text mailers and those used by people with visual difficulties.

I look forward to your considered reply.

Yours faithfully

IPv6 is an important topic, and Steve Gibson pretty much botches it in his Security Now! episode 25.

Now I should add a copule of quick disclaimers: for all the controversy around Steve Gibson (and this is not the Steve Gibson of Truth Driven Thinking incidentally), we should really cut him some slack on this podcast. What he is trying to do on this show is huge, and the breadth of reading he must undertake to understand the issues must not be underestimated. He is bound to make mistakes.

But maybe the problem is that he is trying to do too much himself. He is setting himself up as an expert in all things, but we know the Jack of all trades is the master of none. Certainly there are often large gaps in his knowledge that would be better filled by bringing in some other expert to discuss the issues of, say, NAT or CSMA/CD.

But on IPv6 Gibson’s gap of knowledge is so large that he fails to direct listeners adequately at all. He writes:

If it weren’t for NAT router technology that basically allows many machines to share a single public IP, we really would be in trouble already with IP space depletion. But NAT routers happened, and they’re just a good thing for everybody. Corporations are using them. There are even some ISPs that are using NAT routers and putting all their customers behind a big NAT router because it really works very well, not perfectly, but very well, as most home users know. And so the prevalence and birth of NAT routing technology has hugely reduced the pressure on the move to IPv6.

Steve Gibson is wrong as follows:

• NAT is not a good security solution. The part of NAT that is adding security is the same part that adds security in a non NAT perimeter firewall.
• The gains from NAT have largely been achieved with respect to address depletion. NAT extended IPv4 to give us time to migrate to IPv6, but the gains are not limitless. See the Internet Protocol Journal Volume 8, number 3 for more on this.
• NAT actually doesn’t work that well. We are just getting good at working around its limitations. This is why Gibson endlessly pushes the proprietry non-standard Hamachi solution for encrypted tunnels, and other mechanisms to make some kind of peer to peer work
• IP address depletion is more imminent than the Steve Gibsons of this world think. We are certainly in the last decade of IPv4, and we may see address depletion in as little as four or five years. Again see the Internet Protocol Journal at http://www.exio.com/web/about/ac123/ac147/archived_issues/ipj_8-3/from_the_editor.html

IPv6 has so much more to offer than Steve Gibson realises. Zero configuration, IP mobility, multiple addresses per interface, router discovery, link level encryption (he mentioned that one in passing), authentication… the list goes on.

He also says:

The problem is that it’s not easily compatible with IPv4. The problem that IPv6 is having is, you know, the manufacturers who are making the routers, I mean even, for example, the PC manufacturers are supporting Version 6, though no one’s using it yet. You know, Windows Server 2003 and XP can do IPv6. But you can’t get it anywhere. I mean, there’s nowhere to plug it in to get Version 6

Actually IPv6 does play very nicely with IPv4, and you can get it now. See for instance the BT Exact tunnel broker service.

The real worry here is that Gibson clearly does not understand the mechanism by which we must transition from IPv4 to IPv6. There is not going to be a single big switch over. We must create islands of IPv6 (falling back on IPv4 automatically when we must). We connect these islands by one of the many tunnelling protocols, and as the islands grow, the sea of IPv4 is slowly pushed back. Before you know it we are all using IPv6 - just in time to stave off address depletion.

But whilst the Gibsons of this world stick their head in the sand and pretend this is just not an issue, because we have NAT, we continue to drown in the IPv4 sea.

You want security now? Implement IPv6. Learn how to rewrite your firewalls for IPv6 (yes you need to do that). Learn about its encryption and authentication mechanisms. That is the way to secure networking (well more secure at least).

So in closing - Steve Gibson should keep up his podcast, but until he starts consulting with IT security and networking experts, the podcast will always dissapoint. A pity, as the idea is good.

But I wouldn’t want to do it on my own!

I received a survey the other day asking about my views on the BBC and how would I feel if they raised the license fee considerably to pay for new public services.

We also hear of threats to employers who allow their employees to watch live TV streamed over the Internet, because they must have a license to do so.

Here is my problem with all this: It is already established that if one owns a TV just to watch DVDs and non live programmes, one must have a licence. Why? Because even though no live television is verifiably being watched, the user is capable of watching live TV. Circumstantial evidence is considered sufficient here, despite the fact that non payment of the fee is criminal and potentially punishable by a term in gaol.

So where are we now? If live programmes can be received over the Internet, then all Internet connected computers are capable of receiving live programmes, so presumably the TV license must now be required to own a computer!

That is unworkable (and similar advances in technology in the past were the death knell of radio licences).

The TV licence is an anachronism. It is a relic, and an unjust one at that. It criminalises the least fortunate in our population, denies public services to the most needy, and treats non watchers as potential criminals who must be monitored and controlled, rather than honest people who have turned down that technology.

This is nothing against the BBC. The BBC is a fine and well respected institution, and we should indeed continue funding public service broadcasting for the benefit of all. But it is the way we do it that is at issue.

By my calculation, 2% of the licence fee is immediately lost to the cost of enforcement of the same (this based on figures from Capita group, but it may well be more than this). We could increase revenue to the BBC and right this injustice in one easy move: fund the BBC from general taxation (be it income tax, VAT, additional VAT on TV sets, a tax on content providers or something else).

I don’t think we should make the BBC just another subscriber service. That would lead to further erosion of its public service remit. But with so many of the arts already funded by public money, where is the harm in diverting general taxation to continuing this grand institution?

One would be forgiven for believing that there are no search engines left on the Internet other than Google. No longer do we search for information, we “google it”. Google, through a clean interface, stunningly good technology and a novel search strategy has rightly become the search engine of choice for… well just about everyone.

That is not necessarily a bad thing, but we should understand that there are other ways to index and access information that are better in some circumstances.

Google’s search strategy is a popularity contest. Well linked sites score well. Poorly linked sitres score poorly. New sites, however relevant, need to encourage many people to link to them before they gain visibility in Google. This has also led to a whole new type of spam which has forced bloggers to enable moderation of comments or clever spam filters. For instance, on this site we see bunches of comments that look something like the following (although usually more sexually explicit):

Interesting site. For the best information on Wales, look here: Wales For the best information on a Tour of Wales, look here: Tour Wales

Usually the links number 10 or 15. This is just an attempt to spam the Google index - and unfortunately it works. Unfortunately because it encourages more of this anti social behaviour.

So what can we do? Well there are other search engines, but we need to know how the search engines work before we understand which tool is best for which job. Let me highlight just two:

1. Ask.com uses Teoma. The indexing algorithm on Teoma reverses that of Google. Rather than counting the number of people linking into a site, Teoma indexes based on outward links, and what the outward links link to. Thus for example, an academic site might about snake venom might link to other sites about snake venom. A search on snake venom may bring up the academic sites first, regardless of attempted google spamming.

Of course Teoma is not immune to spamming. One could build a site with many external links and then populate it with rubbish. But this is an alternative strategy at least.

1. Technorati is a search engine that searches the blogsphere. It specifically searches weblogs such as this one and other less pretentious and perhaps more up to date ones, and when there is new news about, it is the quickest way to trak down current information. Other search engines can take days to index information, but technorati will index blogs much more quickly. Of course, if you are searching blog space then you are looking at a bunch of biased sites by definition - but its worth knowing about.

I could mention more - maybe I will in another post, but in the meantime let me ask you: What is your favourite (non Google) search engine? Why? and do you know how it creates its index? Please add your comments.