It is no secret to anyone that the language of the titular nation of our state (Belarusian), although it is an official language, is not very popular among the population of Belarus. Only a very small percentage of citizens speak it. It enjoys popularity only in certain political circles, among the intelligentsia, among some young people and, perhaps, among villagers (in the form of the notorious “trasianka”).
Of course, it is quite regrettable that one of the European languages is, if not on the verge of extinction, then certainly in a difficult position. Finding insufficient support from the state, the Belarusian language lives and develops largely thanks to enthusiasts. And for them one of the ways of self-expression is the internet, where you can write in Russian, in Belarusian, or even in Mongolian, and no one will reproach you: “oh mister, speak normally, I don’t understand you.”
The fact that the internet is a territory of relative freedom prompted the idea of researching how Belarusian-speaking, if one may put it that way, the Belarusian internet is. By the way, there are not so many sites in Bynet compared with the internet segments of other states. Consequently, it is possible to check almost every resource to see which language is used on it. To check, of course, not manually, but by automating the process as much as possible.
Stage 1. Collecting a database of Bynet site addresses
Fortunately, information about Belarusian sites is easy to obtain thanks to various kinds of catalogs, search engines and so on. For the research, the Bynet site catalog from www.akavita.by was chosen. It is one of the oldest and most complete catalogs of sites in the Belarusian segment of the Web. Most of the sites in this catalog are “alive”, but not all of them are updated regularly.
At the moment the catalog was accessed (July 30, 2015), it contained information about 5,315 sites whose administrators considered them part of Bynet and registered them in the catalog. This is of course not the whole of Bynet, but, as statisticians say, the sample is representative. And indeed, it covers a fairly large chunk of the Belarusian internet.
At first there were ideas to use Google or Yandex search results and collect all the sites in the .by domain zone. But for a number of reasons Belarusian sites often have a top-level domain other than the native .by. Moreover, those sites that are aimed at a Belarusian audience and have Belarusian-language content often have domain names in the .org, .net, .com and other zones. That is why it was decided to settle on a ready-made site catalog. Parsing the addresses and names of the sites from a ready-made catalog was not a difficult task, and we will not dwell on it in detail. As a result, a database was obtained on a local computer, and further work was carried out with it.
Stage 2. Developing and testing the language detection algorithm
How do you determine whether a text is written in Russian or in Belarusian? No, with a human everything is clear. Anyone who is more or less familiar with both languages will easily determine the language of a text. Our task is to teach the computer to determine the language.
The first thing that came to mind was to check the text for the presence of the characters “і”, “ў”, and also “и”, “ъ” and “щ”. The first two would indicate that the text is in Belarusian, the rest — that it is Russian.
To test how the algorithm worked, it was decided to go through the site names that administrators fill in when registering their site in the catalog. The thing is that when registering a site on Akavita, the user fills in its name in Russian, Belarusian and English (optionally). It will be rather interesting to find out what percentage of users used the Belarusian language in the Belarusian name of the site rather than duplicating the Russian name. At the same time we will also test the algorithm.
The first run made it possible to determine the language of about 75% of the titles. But the remaining thousand-odd sites, whose names did not contain the letters listed above, also required language detection, and doing it manually was not appealing. Other differences between the Russian and Belarusian languages came to mind. For example, a Belarusian-language joke recently seen on the internet: “ЖЫ, ШЫ пішы з літарай Ы” (“write ЖЫ, ШЫ with the letter Ы”). That is, these letter combinations are a distinctive feature of the Belarusian language.
Having added this and several more rules to the algorithm, I ran the site database through it again. This time more than 300 sites were “left overboard”, sites whose language the algorithm was still unable to determine. After a small refinement, sites whose names contained only Latin letters were filtered out. The rest were checked manually. The language of many of them cannot be established with certainty. For example, a name such as “Футбол” (“Football”) can equally well be attributed to either the Russian or the Belarusian language. And there turned out to be quite a lot of such names.
Stage 3. Summing up the results of the title check
The overwhelming majority of the titles of the sites being checked contained no more than 5 words. But, strangely enough, 48 names contained a mixture of the Russian and Belarusian languages (truly, a country of trasianka). The largest contribution to this was made by the sites of district newspapers. The newspaper names themselves are for the most part Belarusian, while the site titles, written in the form “Site of the district newspaper ‘Зара над Віліяй'”, are precisely an example of mixing two languages in one short sentence.
The results of the analysis of the title languages of 5,315 Bynet sites are shown in the chart.
The table shows the absolute values:
| Language of site titles | Number of sites |
|---|---|
| Belarusian | 1102 |
| Russian | 3826 |
| Mixed Rus.+Bel. | 48 |
| Other | 204 |
| Undetermined | 135 |
Note that the language marked as “Other” is in the overwhelming majority of cases either English or simply the site address written instead of a name.
So, a little more than 20 percent of Belarusian-language names is not bad at all. Let’s see how things stand with the texts on the sites.
Stage 4. Checking site content
After the text language detection algorithm had been tested on the titles, only a small matter remained — to go through all the addresses from the catalog, load the home page of each site and check the language of the texts on it. There was no particular sense in going deeper into the sites. That would have significantly increased the program’s running time, but would not have produced results fundamentally different from those obtained.
Note that a significant portion of the sites (828) could not be checked for one reason or another. These are either inaccessible sites or those whose security settings do not allow unknown robots to visit them.
In addition, on 204 sites the robot found no signs of either the Russian or the Belarusian language. It was possible to establish that sometimes this was caused by an incorrectly recognized encoding. On the chart and in the table the language of such sites is marked as “Other”.
One way or another, it was possible to find signs of the Russian and/or Belarusian language on more than 4,000 sites. This material is quite sufficient to draw certain conclusions.
So, the results are on the chart.
Well, Belarusian looks somewhat sad: 0%. In absolute figures that is 13. That is, only 13 sites where the content is purely Belarusian-language. The table shows this and the other absolute values.
| Language of Bynet site content | Number of sites |
|---|---|
| Belarusian | 13 |
| Russian | 3617 |
| Mixed Rus+Bel | 653 |
| Other | 204 |
| Undetermined | 828 |
To be honest, I expected something more from the Belarusian language. No, of course it cannot compete with Russian, but it could have had at least 10 percent. But, as they say, we have what we have.
One might even say that the linguistic picture on the internet repeats the picture in society. In the end, people speak and write in the language in which they think and in which it is convenient for them to speak and write. One could once again lament the lack of support for the Belarusian language from the state, say that everyone should start with themselves and speak Belarusian. But this is all philosophy. And the reality is that the processes of globalization are now proceeding at a pace they have never proceeded at before. And apparently the Belarusian language, like many other languages of the world, will in time be absorbed by the more powerful language of the neighboring country.
Dmitry Makarski
Machine translation from Russian. Original publication: https://www.kv.by/content/337297-naskolko-populyaren-belorusskii-yazyk-v-bainete