The following are the outputs of the real-time captioning taken during the Tenth Annual Meeting of the Internet Governance Forum (IGF) in João Pessoa, Brazil, from 10 to 13 November 2015. Although it is largely accurate, in some cases it may be incomplete or inaccurate due to inaudible passages or transcription errors. It is posted as an aid to understanding the proceedings at the event, but should not be treated as an authoritative record.
***
>> BYRON HOLLAND: Welcome, everybody. We're going to get going in one or two minutes.
Okay. Good morning, everybody. Thank you for joining us for this panel session. My name is Byron Holland, and I'm going to be moderating the session. My day job is as president and CEO of CIRA, which is the Canadian operator of the dot-ca country and also chair of the ccNSO the country code operator group within the ICANN community.
The session we are having today is on IXPs, but the focus on this is a little further down the life cycle of them, what are the benefits of them, what are the case studies we've seen what are the national we see coming out of IXPs after they've been around for a while.
We have certainly in previous sessions discussed the theory, what are they supposed to do, what are the benefits supposed to be, but today we want to talk a little bit about what have we actually seen and how do IXPs drive connectivity, not just connectivity of bits butconnectivity of technology communities as well as how do they drive local economies, what are the economic benefits of IXPs.
Now, we have a great panel up here with much experience in all things IP -- or IXP, rather. Starting on my left we have Bevil Wooding. He wears a number of hats, but primarily from PCH, I'll say. Mike Blanche from Google, and working down the line, Hernan Galperin, a professor with University of San Andres, Buenos Aires, as well as USC Annenberg.
Then we have Ariel Grazier, who is the CEO and President of The Internet Association of Argentina or CABASE, and Milton Kashiwakura fromNIC.br, part of the host from CGI.br, part of that community. And, as I think many of us who know who are interested in IXPs, Brazil has done some wonderful work on IXPs.
(Alarm sounding)
So that is our panel. What we're going to do is each panelist is going to have five to seven minutes, and I will try to keep them honest on that, to talk about their experience in their region, countries, their communities, and some of the practical and hard evidence that they've seen coming out of IXPs, and then we will have 20 or so minutes for discussion and questions from the room.
So with that, I'm going to make a few introductory -- a couple of introductory remarks, just about our experience in Canada, to set the stage a little bit.
Three years ago, Canada, I would say, was very underdeveloped in terms of IXPs for a number of reasons, primarily historical Telcoreasons. We would have had what I would consider pretty good access to the American Internet, but we didn't really have a Canadian Internet, per se.
And that leads to performance and price resiliency issues, et cetera, as we know that IXPs can help mitigate. So CIRA, our organization, decided to take it on an as activity to help catalyze the creation of more IXPs across Canada. We went from one major IXP, we now have seven that have come to being in the last two years, and they're all a very different story. Similar models but very different creation stories and different stories in terms of where they are after two years.
But a few of the key lessons that we learned is they really have to be community-led. They cannot be top-down imposed, and as much as we were catalysts, we were very sensitive to that, that we had to have the community actually believe in the merits of the IXP and drive it so that when an IXP got up and running, the local community would take significant ownership.
So that was one of the key threads, that they had to be community-led, not top-down imposed.
From a governance perspective, we found that often they're led by technologists that simply want to get going and do the work, but a reasonable governance struck temperature is absolutely key to mitigating problems downstream, so in a sense, getting incorporated, getting a real board of directors, and having a structure is absolutely critical in the very early days to mitigate problems later in an IXP's life. And also the fact that they should generally be a not-for-profit environment to mitigate the costs for the members of an IXP to the greatest degree possible and that the IXP should not be a revenue generator. It is a place to connect, not to generate revenue.
Those are the three -- I mean, there are certainly many lessons learned in creating the five or six new IXPs in Canada, but as far as cross-cutting themes that were relevant to all, and I would posit to any IXP, those are certainly three of the core themes that we learned in the Canadian experience.
Now, with that, I will turn it over to Bevel, who has been in and around this space for a very long time, has worn many hats, and can speak to us in particular from the experience in his region but also beyond. Bevil.
>> BEVIL WOODING: Thank you, Byron, and good morning, everyone. My name is Bevil Wooding. I work with Packet Clearing House as an Internet strategist and as a Caribbean outreach manager. I've been actually involved in the setting up and running of what is now 11 active exchanges in the Caribbean region, either directly or in a support capacity, and that experience over the past five years or so has really given an interesting insight into the different types of start-up processes. Like Byron was saying, the uniqueness that exists in every territory combined with some very common patterns that you see across the region.
So I want to give a very quick history. Most of you would know the Caribbean is made up of some several small markets, and several of those markets have de facto or monopoly type Internet service providers, and when the drive to establish exchange points began somewhere around 2006, there wasn't really any developed technical community in the region. The initiative was really championed by the CaribbeanTelecommunications Union. I'm still there in a support capacity for the Caribbean Telecommunications Union, and we felt at the time given the emphasis that was being placed on the Internet and Internet economies and knowledge society, this whole issue of what's the foundation that allows for the creation of this knowledge-based economy, what was that foundation? We felt that exchange points had a big part to play in it. I remember Bill giving a presentation in Curacao in 2006, and the response of the regulators and government officials at that meeting was why hasn't it been done anywhere in the Caribbean? Surely there's some secret that you're missing out or some bad story that you're not telling us about, all of these benefits or IXPs are supposed to provide.
What that did back in 2006, really, it set the track for what became an approach to IXP promotion that I have not seen or heard about in other parts of the region. Most of the IXPs that have been established in the Caribbean have been regulator-led, not regulation policy-led but regulator-let. The ISPs, that's service providers, actually requested that the regulators in most countries -- I'm talking about 90% of the exchanges -- that the regulators play a mediation role in bringing them to the table to discuss what should have been an obvious technical efficiency that would benefit their networks. From cure sow to where we are now, the regulators updated a key part in promoting the exchange points, but what that has done is created a situation that now as we have, as I said, 11 active exchanges, the IXPs having fulfilled this requirement of having an exchange point, have not committed sources to continue to the growing of exchange points, and so we have a situation where we can look to see territories where you have a local champion, somebody who is both aware of the technical requirement for exchange point but is also keenly aware of the need to encourage local content development or encourage the situation of needed foreign content to the local market.
Those are the ones where we've seen growth and development, and the ones where IXP simply fulfilled the regulatory or government pressured requirement are the ones right now floundering.
In the region we've gotten the same kind of pushback that other territories -- dominant players are saying we don't need this, there is no local content, these markets are too small. We even had reasons as to why IXPs were not necessary for the Caribbean, such as it could negatively affect government tax revenues if they disrupted the traditional income flows of the income-owned operators, to statements like our networks are already optimized to deal with the traffic consumption patterns in the region.
And what we had to do very early that I think led to the momentum that we gained, what we had to do very early was move that conversation away from the technical actors and broaden it to bring in the content creators, government agencies that were seeking a fast or cheaper Internet, and so the working group exchanges started to look like a multistakeholder environment where people were more interested in the benefit that they were not now seeing from the Internet than they were in what was the technical underpinnings for it. That created a very important platform for what we have now, and it also, I would say, provided us with some important counter points to the kind of antiIXP sentiment that was expressed.
One, of course, was the general awareness that every country should have a sufficiently robust local Internet infrastructure, and I'm saying this, and they may seem obvious, but for the government officials, for the private sector, these were not obvious follow-on points or benefits from establishing an exchange point. Remember the discussion up to that point were amongst IXPs that were enjoying a particularly closed economic circle.
Once we cracked that open, this issue of what are these better, greater economic opportunities started to come to the fore.
So we're at a point now where one IXP has said openly in a public forum that we know what we're doing is inefficient, but no one has asked us to stop, and if you come to the regions, the territories in which we operate in and push for an exchange point, we will support you, but if you don't, we'll continue as we are, and I think that kind of summed up the general disposition of the common operators, so we're at the point now to build that momentum, we have to establish an effective technical community, so CaribNOG, the Caribbean Network Operators Group, started out of the real addition of the need to have an active community around the exchange point process. Our research, we brought the universities in to ensure that they can benefit from these new hubs for monitoring and observing traffic pattern and the like. Content, bringing in the not just government actors but local media houses and allowing them the understanding of how local traffic to local audiences can improve their business models, and of course, stakeholder engagement, and next year we'll be actually looking at a State of the Caribbeanbroadband connectivity plan to get real figures about the difference between the broadband that is being promoted or marketed and the actual speeds being achieved by consumers. All of this is to give further strength to the argument that local Internet infrastructure is a necessary part of building out an effective economy.
So summary, champions, local, important, governance, absolutely clear -- a clear sense of what the exchange points need to do for them to be most relevant in local locals, a -- local environments, a local strategic plan, how you're going to appear to local content, and regional community engagement as a key forum for not just initiating the process but continuing it onward.
>> BYRON HOLLAND: Thank you, Bevil. A very interesting story where you had to actually educate many stakeholders on the benefits and not just on the benefits but mitigate the fears.
I thought also one interesting point made there was the notion of educating the content creators. When people think of IXPs, that's often not one of the communities they're thinking about, but if you can educate the content creators on how to get their content more effectively, more efficiently, to their local market, they can become a very loud advocate.
Okay. On to Google and Mike Blanche.
>> MIKE BLANCHE: Thanks, good morning. My name is Mike. I work for Google in London. I do a variety of things working on infrastructure development with Google, building our network, working with our Telecon partners mainly with Europe and Africa, and have been involved in a number of Internet exchange projects in different places. I'm also a director of the London Internet Exchange, but I'm not wearing that hat today.
So the first thing to say is Google is definitely a big fan of Internet exchange points. We're a member of 70 IXPs on every continentaround the world apart from Antarctica, and when the penguins set one up, I'll probably go and join that one. We've supported the establishment and growth of Internet exchange points worldwide, both directly and through organizations like the Network Startup ResourceCenter, who do an amazing job.
We see Internet exchange points as really an important part of the Internet ecosystem, and it doesn't matter whether it's a large one, a new IXP, or old IXP, that they're all important and they all bring benefits. Bevil talked about a startup and growth phase of IXPs, and I'd like to talk about the impact of that, of how Internet exchange points can help transform economies into Internet economies.
So I think there's three kind of categories and orders of effects that happen when you have Internet exchanges. The first order of fact is sort of the simple ones that most people know about these days, that when you pair it, it reduces costs for the people you're pairing with it, improves performance by reducing latency and so on, and it brings -- operates together into a community.
The second order of fact is on the broader Internet infrastructure in a country and in a region. It enables new infrastructure, new connections, and new content to be hosted in a region.
And the third order is what we're really talking about here is the Internet economy part of it. It enables new users to come online, it enables development of new services, new revenue, new business models, and this whole new Internet economy that we're interested in building.
So the first type of effect is the simple stuff, the act of interconnecting, of connecting your network to somebody else and exchanging traffic has benefits that we all know about and have been written about many times in terms of improved performance, reduced costs, mutually beneficial relationship between the people that are connecting.
We can help build that by building the network effects of the Internet exchange point by getting more people to connect. The more people connect, the more benefits there are, and the people connecting can come from all parts of the Internet ecosystem, it doesn't just have to be telecoms operators and IXPs. It can be content companies, governments, educational networks, and everyone has a part to play in helping to increase those network effects.
The second type of benefit comes from this new infrastructure, and what you start to see when an Internet exchange point starts to build scale is that other interesting things happen, new submarine cables start getting built to countries or national long-haul infrastructure starts getting built. We saw this in Eastern and Southern Africa when the Internet exchanges in Nairobi and Johannesburg started to grow and get scaled. Companies like Liquid Telecom built the first link between those two cities crossing four or five countries, so you no longer had to use submarine cable capacity to reach between them. And today over 100 kilometers of fiber optic cable is built and buried in Africa every day. That's not down to IXPs, but there's an influence there.
More infrastructure comes with data centers and colocation facilities. Now, there's an Internet exchange point where there's a place where content providers can come and store it very close to where the users are, and carriers, each with data centers, have started to spring up in Africa, in again in Johannesburg, Nairobi, Mombasa, than other cities around Africa, and an Internet exchange point is a real kind of magnet for those kind of developments to happen.
And then you start to get new transit providers coming in, companies that previously would never have connected to these countries bring their services to a country, and quite often that results in the price of transit falling and the cost of connectivity falling because there's new providers offering services. And when the transit providers come, quite often then the content providers come as well, and Google and other content providers are big investors of infrastructure. In the past three years, content providers have invested over $100 billion in infrastructure around the world in terms of data centers, submarine cables, network providers, content delivery networks, and bringing it close to where the users are is a key part of that investment. There's a virtual circle you're starting to get.
The third type of impact I think is the most exciting, and this is what we're really looking for. With these reduced costs that you get, you get new users coming online, people who previously couldn't afford to be on the Internet are now able to come online. There's new revenues for telecoms operators even. There was a study by the Internet Society in 2012 that looked at the impact of the Nairobi, Kenyan Internet exchange point, and even then they found that -- they estimated that the telecoms operators in Kenya generated $6 million of new revenue just because of increased performance, so users used the services more because the performance was better because they were an Internet exchange point. This was three years ago. These benefits I'm sure have grown many folds since then. And there's new services that previously wouldn't have worked that now are enabled.
So for example, e-Government services, online tax filing, online education, and finally new businesses. If you could reach your local market and the people know -- that know the market best are able to develop applications and services that are relevant for that market and deliver them efficiently, cost-effectively, and with high performance, that really helps build this new Internet economy.
>> BYRON HOLLAND: Great. Thank you very much, and some really interesting not just anecdotes but real-life stories about benefits that we don't necessarily think about when we're thinking about the benefits of IXPs, so those -- I mean, just the fact that -- I know it's not a direct result of but that there's 100 miles of fiber being built in Africa every day is staggering.
To just actually provide some anecdotal evidence to back up some of those comments from the Canadian perspective, two IXPs went in, in particular two markets that were very underserved had incumbent operators and everything that goes with well-established incumbent operators, but once the IXPs got started, hearing many of the reasons they shouldn't, but once they did get up and running. A global transit provider, we were able to convince to join, and they saw the merits of it, and Hurricane came into those two markets, Hurricane Electric, where they never had been before, and we saw the cost per Meg for IXPs at a wholesale level drop between 15 and 20 Canadian dollars per meg to $5 per meg within months, and in fact, now in one of those markets it's about $2 per meg, so when you think about over the course of 24 months the wholesale cost going from 15 to 20 to 2 as a direct result of starting the IXP, transit being attracted to that, and just recently a content provider located there, so the virtue of cycle that Mike just spoke about actually happened in those two particular IXPs.
And with that, Hernan.
>> HERNAN GALPERIN: Thank you. Mike summarized incredibly well the case for IXPs. Now -- and we -- I'm a researcher, and I come with a bit more of a skeptical perspective, say, well, the theoretical case is very well established, and we all, I believe, the great things that could happen and should happen are great, but are they really happening? And this is the question that we set out to study in Latin America, and particularly trying to establish counter-factuals because Internet is evolving so fast, prices are dropping, and traffic's increasing, so the million-dollar question is these trends, would they have continued or has the IXP accelerated and by how much the IXP's really accelerating these trends that are in themselves already happening? So we tried to bring some more evidence to the discussion by studying Bolivia, and Bolivia is a fascinating case, and we chose it for several reasons. One of them is that Bolivia faces some of the most biggest connectivity challenges in the region, just to give you a brief sense. It's about 17% connectivity in terms of regional average, there's only 17% of the population online or however you measure it in terms of the regional average. Now, Bolivia is a poor country, but even if you take into account wealth and so on, it's way below from what you would expect in terms of its level of wealth.
It's also where prices are highest. We measured prices in Bolivia, access prices at the retail level, they're about three times higher than the region level in Latin America, which is already very high, and part of the reason that prices are so high and volumes are so low in Bolivia is it is, as you know, a landlocked, and it's -- the transit prices are incredibly high. We found prices always over $100, $120 a month, where in Lima and Santiago, where -- which is where the traffic was going through, you can find prices for about $20 or $25, so clearly there's a markup of four times between just the traffic from Bolivia into the undersea cables in the Pacific Coast.
So you would think that with all the situation with low volumes, high transit prices, IXPs would be an ideal and obvious solution to some of those problems.
And the second interesting thing is what the government did, which it passed a new interconnection law in 2013, and the law explicitly said we require that all ISPs come together and create an IXP, and the law initially said you're welcome to create it however you want it, just do it, and they couldn't agree on how to do it, so the government had to come in and basically establish the parameters of how the IXP would work, and I can discuss -- we can discuss a bit more of why it wasn't that a great outcome of the process.
So we set out to establish where the impact was of the IXP in Bolivia, and one of the things we did, which is we deployed probes, network probes around the edges and the network of Bolivia, and those probes were basically sending trace routes to different -- well, to all/24 prefixes in Bolivia, and we left them for several months, and we measured two parameters, we measured run trip time and the number of hops in those routes, and we then categorized the routes between international routes, basically routes that we knew were going out of Bolivia and then coming back, which is the usual (Inaudible), and then routes that were going now through the new IXP, and so we compared the results in terms of the round-trip time and the number of hubs for those two categories of routes.
So we found great results. On average, on average, the great that's going through the IXP was, in terms of latency was 70% below that of an international route, so you have a gain of 70%, 70% of lower latency, of lower round-trip time for a route that is within IXP or passing through IXP compared to the route that is going out and inside of Bolivia, and we also found a reduced number of hubs, about a 30% reduced number of hubs in a route going through the IXP as compared to the international route.
So the technical case seems to be compelling for the IXP in Bolivia. Now, the reality of the IXP in Bolivia as of now is that it's stagnant, that it has not increased.
And so briefly, where we think the lessons are is first that it's not enough to have a technical compelling case. There has to be the economic incentives and the political incentives have to be aligned, and this is where we come back to the idea that the government basically had to impose the IXP model on the ISPs, and the ISPs, for a number of reasons, were less than happy about that. And also the economic case has not been compelling, and I look at it as a problem. Because the traffic volumes are low, the city ends are not attractive to the IXPs, and since the city ends are not attracted to the IXPs, traffic doesn't grow and you get into a cycle. So also the economic case, which seemed their theoretically compelling has not been realized in Bolivia.
And lastly, the lesson we drew was the idea, I think, Byron, you mentioned, the top-down, bottom-up. This was not a bottom-up process. The government initially wanted it to be a bottom-up process. The IXPs did not agree. The government had to come in, very top-down, established the very detailed rules about how the IXPs would work, and in fact, hosted the IXP within the state telecom operator of Bolivia, which created even more problems for the IXPs. So, really, the lesson is there's not enough to have a technical compelling case, this has to be a good alignment between economic and political incentives for the IXPs to flourish. Thank you.
>> BYRON HOLLAND: Thank you. And I think a very important case study in potentially the challenge of doing -- you actually lived the experience of imposing it down and not having it work effectively or live up to the theory. Thank you.
Ariel.
>> ARIEL GRAZIER: Okay. Good morning, everyone. Let me share with you the Argentinean experience and try to set a couple of points because what Mike summarized just before, I totally agree, and we also fight with the problem of Argentina is the last country, and we have very, very several regions without connections, without connectivity, so we needed to solve the problem, so we create a model starting in 1995. We started our first IXP in the capital of the country, Buenos Aires. And at that age we pay just two gig the numbers. We pay $40,000 for 64K connections, $40,000 monthly, okay.
In 2004 we get at the IXP all the telecom companies, all the ISPs, and we can get in the IXP in the Buenos Aires no more than 1% of total traffic in ISP needed, so after that, we start a process. We started in 2009 a process that we named -- the way to get all the cities -- the mayor of cities in the country at the same level, okay. It's federalization of the bandwidth, we named it, and so in that process we understand that in the case of Argentina we needed to interconnect our IXPs because if not, we will have the same case that I understand is happening in Bolivia. They don't have the ability to grow enough, so we build out a process to open more than the Buenos Aires one, and we're still doing. In fact, last week we opened the number 16, that is up and running now, and the grow of the traffic and the drop of the cost are the numbers -- the magic numbers that we get. One ISP or one connectivity provider or one content provider that is connected to one of our IXPs because all our inner network can get between 65% to 72% of the traffic they need than the one we have ten years before, so -- and the traffic and the cost of the traffic drops to $4 in Buenos Aires, or $18 in a very far away city in the north of the country more than 2,000 kilometers from the capital, and the cost -- and the reason of these different costs is because we need to pay the national long haul to reach the other IXP, the northest one, so bad.
Just to share with you the numbers, they have been paying the last month in October $400 for a meg, and now they are reaching the 65% of all the traffic at $18 per meg, and the magic of the number is not just the price of the meg, it's the viability of the bandwidth, and this is the difference.
When you open an IXP, the availability of the traffic and the bandwidth is the difference. Then the prices goes down, and then it's a virtual -- a virtual sequence that you grow and grow and grow because -- but the reason why we do it and we are facing this is because it's a national issue for us to interconnect and to develop new ones. We are a nonprofit association, but we take this project as a national project, and I guess this was the difference in different case, for example, with Bolivia and in other cases in Latin America where they're not connected or there were different model, like in Brazil, which is -- they are sponsored by the CGR or the NIC.br. We start as the ISPAssociation. Now we changed to the Internet Association because we accept governments, we accept universities, we accept content provider, we accept media providers because they give us local content, local traffic. They are very important key participants and members of our IXPs.
Just to share with you another important issue that we changed in our history is that at the beginning when we started with the first one, we require for the telecommunications government license to be connected to the IXP, because we understand that was just a telecommunication issue. Now we say that it's not, so we cut this mandatory issue and we changed that it's open. Everyone that have content or traffic to put on the -- on interchange on the Internet is welcome, and I think that this change of mind on every of us, including the government, to say that, okay, you don't need to be a telecommunication operator or under the license to have content and provide -- be interconnected in an IXP, so I guess this is part of the story that --
>> BYRON HOLLAND: Thank you. That's a very powerful story and interesting to see how it was an evolution in terms of thinking of it as just a group of ISPs exchanging traffic to an open, inclusive, anybody with content can participate. Also that part of your plan included connecting IXPs as individuals to make the whole thing work and, of course, the number that's remarkable is $400 a meg to $16 a meg.
>> 18.
>> BYRON HOLLAND: $18 a meg. Milton, the story in Brazil.
>> MILTON KASHIWAKURA: Okay. I'll try to share our experience. In our case, in Brazil, Internet exchanges started in 1996, so this was a Internet exchange, so these academic exchange grow or does Internet exchange from the commercial institute start, so -- because this academic model, we use it to obligate every participant to exchange traffic to every -- all these participants.
So mainly the (Inaudible) didn't like this kind of model, okay. So we have -- we have some from the data center, from the associations. Because of the growth of the -- these academic interchange exchange that we have for -- this was very populated, they don't like to continue this kind of project, so made an agreement which is the -- they pass these kinds of issues. The Brazil International Committee see that this is bad things because we needed to have no commercial Internet exchange in the country, so we started our projectPTT, PTT Metro project in 2004, okay. So we heard from -- we heard from the PCH that the model that you have more economic model is a model that interconnected all Internet exchange that you have in one measure, okay.
So we started the Internet project, inviting all the Internet exchanges to interconnect all together, and in many market Internet exchange points at that point, okay.
So we started this project totally open, all participants can do everything, only they need is strategic trafficking in IP, okay.
So we work a launch, mainly because in that moment, there some who would like to participate, so we heard that presented in our meeting a model. I don't know if you know the (Inaudible) model, and the others IXPs is in the dot net, okay, so to have the -- the traffic from IXP to the other, we needed to pass and go to the -- so the point modify this kind of picture. So nowadays I agree that the totally traffic don't pass through the Backbone because of the Internet exchange doing this kind of service all Internet service provider in the country.
So nowaday we have more than 900 ASC interconnected in our project.
We started the project with the Brazil -- Brazil has only 177 AS in 2004. Now we have more than 3,600 AS in the country. About 70% of AS participates in projected account. Nowadays the ISPs, the medium in the model, IXP in the country have presented about 25% of the access in the country, okay.
These participants share with us about 60% of the traffic come from the Internet exchange points. Mainly the IXP that obtained has household access. When the ISP obtained the commercial access, about 30% came from the Internet exchange point. So this kind of infrastructure affected the cost of the Internet service provider in the country.
About -- the -- to participate in the Internet exchange point from these ISPs is mainly because the cost of addition and better quality of infrastructure. After the connection, the feeling that these ISPs feel in this kind of infrastructure is that the qualities is very good, so this is our main reason to participate in the Internet exchange points. After --
>> BYRON HOLLAND: Milton, in order to make sure we have time for a question, I'm going have to ask you to wrap up fairly shortly.
>> MILTON KASHIWAKURA: Okay. Thank you. So I needed to try this kind of -- the other services that you -- we put in our infrastructure, like quality of the measurements, servers, the ADTP servers, services like this that has a good quality of the Internet outcome. Thank you.
>> BYRON HOLLAND: Thank you very much. Interesting how often the exchanges can start in such different ways and that yours came out of research network and academic network, and often the traffic of exchange networks is a good base upon which to build. Well, which -- and correct me if I'm wrong, you said 60% of traffic is now flowing through IXPs in Brazil. That's quite a testament to the success of them.
Thank you very much, everybody. And now I'd like to turn it to the floor. Any questions? I mean, we have a pretty remarkable range of experience here, not all of it successful outcome, which is often where the best lessons are learned. Any questions for the panel? Sure, you just have to come up to a mic. There's two stand-up mics in the back.
>> AUDIENCE: Hello. My name is (Inaudible). I'm from India. I work for National Internet Exchange of India, and now we're also having a same, similar problems that Argentina faced a few years back, like what we are having a problem with is we are having an IXPs, but the traffic is not going up because we are -- as further IXP license criteria is there to connect to the ISP, but we are planning to connect to content provider. My question is the interconnection of the ISPs will make us transit ISP kind of thing. We will become an ISP again, so how -- but how it will work, actually, so because somewhere -- and I saw some I studied, there is a model also to connect between ISPs.
>> BYRON HOLLAND: Who'd like to take that one on? Ariel.
>> ARIEL GRAZIER: Yes.
>> BYRON HOLLAND: Okay.
>> ARIEL GRAZIER: I think that it is an issue also with the time of agreement, and I think it's part of the thing. We have set up at first the multilateral agreement, which means everyone is obligated to interchange all the traffic, and the incumbent doesn't like it, and the transit IP providers international, they don't like neither.
And the thing overall is to find a champion, and not -- not a person, a company, an IP transit that wants to go with that, and after that, the work is to allow within. In part of the negotiation, we allow to have bilateral agreements, not only the multilateral agreement. The multilateral in our case is mandatory. The bilateral is an option, and this was the key to have the other IP transit, and also we defined -- made some tuning, if you want, that in the multilateral agreement, you are obligated just to put in the traffic to push the traffic, your original or local traffic, not your international traffic, not the traffic that you sell for the others, and I guess maybe this can help.
>> AUDIENCE: Okay. So your IXP charge for --
>> ARIEL GRAZIER: No, we have a very -- we are for-not-profit, we just cover our cost and charge our members just to cover the cost of the operation of each one.
>> AUDIENCE: So that is less charges you are charging for 10 megabit.
>> ARIEL GRAZIER: We charge part for the port and part for the other expenses we have. It's very cheap, it's very, very cheap, our model.
>> AUDIENCE: I --
>> BYRON HOLLAND: Sir, I'm going to just -- because of limited time --
-- you can tell there's a wealth of expertise here, so I would suggest that you take your further questions offline. I'll go here and a question here.
>> LEONID TODOROV: Thank you. Leonid Todorov, general manager APTLD. As you know, some of our members classify under the most challenging categories, which is landlocked, small Pacific islands, and least developed countries.
Now, in some of those territories or countries a local ccTLD is at the same time a registry, let's say, a regulator. The only source of -- well, ISP, a regulator registry, I would put it in such a way, so if there is any lesson for them from you, I mean, speaking of collective wisdom, any lesson to learn from your experiences and any recommendations as to how to develop ISPs, especially considering that, for example, between two islands there might be, like, 4,000 miles of ocean and no submarine cables and no infrastructure in place, and of course, great vulnerability because of all these, you know, natural disasters. Thank you.
>> BYRON HOLLAND: Well, that's a tough one.
(Laughter)
Who would -- Mike.
>> MIKE BLANCHE: So that's an interesting question. I can talk about an example I know a little bit about, which is not a small island but is a small landlocked country called Lesotho, which is in South African, surrounded by South Africa. They have three telecom operators in Lesotho, and the regulator got involved and got something set up to help exchange local traffic locally, and this -- even though it's only between three operators, which is the minimum you need for the Internet exchange -- actually, you can't have less than three because then it's just a connection -- between three operators and it's only exchanging a few megabits of traffic, given the price for their international connectivity up to the time I was talking to them two or three years ago, was $1,000 a megabit. A few megabits of traffic between the local ISPs has a big impact.
Also to the other gentleman from India who says he only has a small amount of traffic on his Internet exchange, it's not necessarily growing, think about what the value of that Internet exchange is independent of the traffic volume, and there can be a lot of direct value. And then we talked earlier about some of the indirect value and the second-order effects that come from that as well.
So if you can -- if there's multiple operators you can bring together on these items and even if it's only very small volumes of traffic that are exchanged, there can be a lot of benefits there.
>> BYRON HOLLAND: Thanks.
Gentleman here. Sorry, did you want to add something to that?
>> ARIEL GRAZIER: No.
>> MIKE JENSEN: Thank you, Byron. Mike Jensen from the Association for Progressive Communications. I would like to ask for clarification on this idea of having mandating multilateral peering and having bilateral peering. If you already have mandating multilateral peering, it's not clear to me how the bilateral peering fits into that.
The question I had was around, I guess, scaling down IXPs. It's very interesting to see the members of the panel here who have experience with many IXPs in the country, how far down do we go in terms of the small villages that are going to ultimately have IXPs? What are the dynamics that drive the duplication or multiplication of IXPs within the country, thank you.
>> BYRON HOLLAND: Ariel, did you want to speak to the first question, and then anybody else speak to the second question.
>> ARIEL GRAZIER: In our process, the multilateral agreement was as a result of the incumbent policies. At the beginning we didn't have. As the incumbent leave us and trying to sell IP transit, we built this kind of shield saying at least you are obligated to give us your local traffic, and this is the multilateral. And then when we understood that, the traffic didn't grow up enough as we needed, we understand that at least one IP transit, international IP transit must be part of us, and we understood that they need to sell to the others part of the other traffic, the international traffic, so our history was -- or we made some kind of economic pool that we brought all together from one more capacity or we allowed them to have this kind of bilateral agreement. I don't know if --
>> BYRON HOLLAND: Anybody want to comment on how small can you go, how small is still viable and worthwhile?
>> BEVIL WOODING: I can give a couple of examples using British Virgin Islands, one of the first IXPs in the English-speaking Caribbean, with a population of 32,000. We had three operators, all of whom had very profitable telephony businesses and all of whom resisted starting an IX until the regulator did this one simple thing, said that any local traffic should remain on local networks. Didn't specify IXPs, didn't just any solution, just made that edict, and once the exchange got running, got up and running, the providers found that there was, in fact, local traffic to be exchanged. Now, you're talking a population of 32,000, so we're not talking massive amounts, but given the wholesale amounts of bandwidth, it turned out to be a significant figure for those IXPs. And when Grenada started, population 108- OR 107,000 and St. Vincent, which was most recent, they found that even though the populations were small and networks were moving huge amounts of traffic, the cost saving per megabit kept in country was significant enough for them to reinvest in infrastructure development and so on, so I have found that size does not, in fact, matter -- population size doesn't matter. Keeping local traffic is what matters.
>> BYRON HOLLAND: Thanks. That's a great insight. And there was -- last question.
>> MWENDWA KIVUVA: My name is Mwendwa Kivuva from Nairobi, Kenya. A lot has been done to establish IXPs, especially in the developing world, and ISOC and Google have set up for these IXPs. What I'm wondering is (Coughing) excuse me is what can be done to ensure at least one IXP in each country because we have been on the ground for ten years, but the problem is it's so small in the growth of these number of IXPs. For example, in my region. What can be done to -- because we all know the benefits. Are preaching to the converted, but we still don't have these IXPs in our regions, but Kenya is an exception. Thank you.
>> BYRON HOLLAND: A simple question, very hard to answer. Anybody want to take a very short stab at answering that? Mike, sure.
>> MIKE BLANCHE: So the Internet Society did a great job in Africa with an access program on training people how to set up Internet exchanges. I think the key point to wrap up is you can't force things, and I think Bolivia is a great example of how forcing an Internet exchange doesn't provide any benefits, and Internet exchanges is a -- to solve all your economic problems. If you have all the basics in place in terms of the right laws, Internet exchanges will spontaneously appear. They can sometimes have a bit of a push and a help, but we need to get the full environment right for these things to happen.
>> BYRON HOLLAND: Thanks. And actually, that captured probably some of my final comments to wrap this up. You know, we've heard a couple of stories about how they've started and successful ones have started. There's a unique story to almost all of them, whether they start with research networks or prodding from regulators or spontaneous combustion in some other way. There's no single story to get them started, but it typically does, as we heard -- it takes a champion. Sometimes that can be an individual. Most often it has to be a company or organization who has the muscle and the interest, and I think that speaks to the next point. There has to be an interest and not just I'm interested in doing this but a commercial rationale for them to be sustainable that each of the actors and participants see for themselves, and while they may not see it at the outset, part of what I think I've learned in our journey and I've heard on this panel is that it takes time and effort for the actors to actually see their benefit and to not necessarily worry about that, to appreciate that it may take some time and education for the actors to actually see their benefit, but generally they do, be it price, be it performance, resiliency, redundancy, whatever. The other interesting thing here is all of the spin-off benefits that we may not necessarily associate with IXPs but that are real, be they fiber being laid, new data centers, et cetera, as we've heard, and ideally, it takes many different types of members, not just ISPs, but also content providers and other types of members and an open, inclusive IXP is the kind of IXP that is most likely to succeed in the long-term.
But it's not all roses. Just because you start one doesn't mean it will survive or show benefit to the community, and it takes many factors to come together to create a successful one, and I think we've heard an interesting story about how sometimes they don't work or can't work or they go into hibernation. And I know in Canada we've experienced both success and hibernation in our story, and as Mike said, they're not a panacea, but they can provide a huge benefit to the community if they're done right. I'd like to thank all the panelists for their wealth of experience and thank you for the questions. Thank you, panelists.
(Applause)
(Session ended at 1208)