Teaching Morse Language to a Deaf-Blind Person for Reading and Writing SMS on an Ordinary Vibrating Smartphone

Andras Arato1, Norbert Markus1, and Zoltan Juhasz2

1Wigner Research Centre for Physics, Hungarian Academy of Sciences, Budapest, Hungary

{arato.andras, markus.norbert}@wigner.mta.hu

2Pannon University, Veszprem, Hungary

juhasz@irt.vein.hu

Abstract. Deaf-blind people have a very small window to the world. New technology can help, but portable Braille lines are expensive. We developed and tested a very low cost method for reading and writing SMS messages with a Hungarian deaf-blind person using Android smartphone with vibrating motor built in. Words and characters were converted to vibrating Braille dots and Morse words. Morse was taught as code for recognizing characters and also as language for recognizing words.

Keywords: Deaf-blind, Morse Language, Language Teaching.

Introduction

PG was born deaf and became blind when he was 45. He learned reading and writing Braille and also to send SMS messages on his simple phone with no feedback. Until recently he was able to read messages only with the help of his deaf wife.

A straightforward solution would be a haptic touch screen but this is not yet available [1]. PG knows several languages such as spoken Hungarian, Hungarian sign language, writing block letters in the hand, Lorn and Braille.

The Extended Definition of Language

From the point of view of learning we can give the definition of language as follows: It is for communication among people. The community speaking the given language has an identity. The perception is accomplished with some of the human sensory organs. The reproduction is done by a motoric program. The perception and the reproduction are controlled by a common, hierarchically structured mental dictionary. [3]

In spoken language, the perception is accomplished with hearing and the reproduction with speaking organs. In sign language perception is haptic in the case of a deaf-blind person. The difference between the spoken language and the Morse language is minor. The speech has three main characteristics: the rhythm, the energy and the frequency, the Morse is characterized only with rhythm. We can interpret Morse as extreme speech. Deaf-blind persons can percept vibrating Morse with their hand.

Hearing people perceive Morse language with their ears. Before they have to learn the “phonetics” of Morse characters, but they also have to learn whole words and phrases too. Hearing people learn spoken languages in a similar way. We have to build our mental dictionary which is a hierarchical structure. Deaf-blind people do the same with vibrating Morse.

Teaching Morse language we found, that different speeds must be learned separately. If somebody learns fast rhythm he or she has to learn slow rhythm too. PG had similar difficulties with vibrating Morse. See later the problem with “m” /--/ and “i” /../ in the next chapter.

The Process of Learning Morse Language

The most difficult part of creating the assistive device for PG seemed to be the task of teaching him the Morse language. We embossed for him the Hungarian Morse codes in Braille. We introduced a new code for the Space = AS /.-.../ (the code for wait in international Morse code), since the usual pause cannot be used in single-character cursoring mode. For simplicity we only have two levels of cursoring, the character-by-character and the word-by-word level.

PG learned the codes in a few weeks, but it was only a theoretical or “dry” knowledge. Learning language is effective in live practice, so we modified our Mobile SlateTalker (MOST for Android) program's text entry mode [2]. Text entering is done in Braille dot by dot. A single vibration indicates successful dot entry. A double vibration indicates clearing the Braille dot. Our deaf-blind learner could enter characters in Braille accurately after a few minutes of practising. He was motivated with this success very much.

Recognising Morse characters by the phone's vibration proved more difficult for him. Facilitating this process we introduced a control for changing the code speed. We suggested him to learn at a relatively high speed [4] and finally we found 15 characters/minute adequate as a general speed. Now he uses 10 characters/minute for unknown texts and 15 for known ones e.g. reading menu words. He also asked us to help him distinguish Dit-s from Dah-s, so we changed the weight from 1:3 to 1:4.

As a last resort, he has a haptic Braille support when reading in character-by-character mode: he can check every letter dot-by-dot in Braille mode. Touching any one of the dots of the Braille cell it vibrates if it is „raised”.

It was an interesting experience when he first changed from a relatively high Morse code speed (15 characters/minute) to a low one (10 characters/minute). He recognized the code for “i” /../ as “m” /--/ several times. It underlines the correctness of our research about the necessity of building the mental dictionaries for different speeds [3]. We strengthened the motoric-control side of PG’s Morse language knowledge using a relay buzzer and Morse key. The relay buzzer was not found satisfying, so we finally used vibrating motor with the Morse key.

In this video [7] you can see the learning process with PG using a Morse straight key and a vibrating motor. PG has problems understanding unknown Morse words. He asked to slow down and increase pause between characters sent. He recognizes his name sent by the first author, and answered sending his name. The communication between the two persons is done by speech and writing block letters in PG’s hand, arm or shoulder. Sending with correct rhythm is always emphasized.

Developing the Special Version of the Mobile SlateTalker Program

We chose an ordinary Android phone, installed MObile Slate-Talker on it and placed a special Braille mask over its touch screen.

We decided to enable MOST to give haptic Morse feedback in the menus and for reading texts by character, word, sentence, etc.

A special haptic feedback has been added to facilitate entering Braille cells dot by dot accurately.

A so-called haptic Braille touch exploration mode has been developed to allow checking each individual Braille dot of the currently cursored character.

The Morse code speed may be changed any time on the Braille mask.

The Morse code weigh can be changed at compile time.

As a useful side-product, the new features may also be beneficial for the hearing blind users in extremely noisy or very quiet environments.

The entire MOST program with a fully-fledged haptic support seemed to be too high as an entry level for PG. We decided to create a special restricted program environment for him within MOST. In this environment, he only finds things he has acquainted with and he only has to use the skills he has already acquired during the weekly Morse sessions. As he learns something new, that particular feature is added very soon to this deaf-blind mode.

In this restricted environment, PG cannot get lost, and cannot activate functions unusable for deaf-blind (e.g., answering or placing phone calls, starting audio playback, etc.). He is prevented from venturing out to the uncharted waters of the MOST program and the Android system. Currently, incoming calls are immediately rejected and alert dialogs that happen to pop up accidentally are immediately dismissed programmatically.

The main turning point came in late December 2013. He was allowed to take home the specially programmed Android phone and could read and send SMS text messages all by his own. We witnessed (and still do) numerous occasions of his pleasure over his newly gained autonomy, as he gradually realizes how much he can achieve with an ordinary, off-the-shelf Android smartphone without the need of constantly relying on others' assistance.

Each week, PG comes up with a new demand for a feature he wants to be added. This is how just recently an intuitive method of easily replying text messages got built in the special environment for deaf-blind, and his ideas and demands keep coming. For instance, his latest assignment for us is a haptic alarm clock. A recently introduced new feature includes reporting the current time, date and battery charge level by haptic Morse digits and delimiters when a dedicated Info button is pressed.

In this video [5] PG is looking for the requested name (A András) in his phonebook. He can read the known names in Morse with a reasonable speed (15 characters/minute). You can hear the buzzing of the vibrating motor of the smartphone. The text-to-speech output is only used for debugging purpose. After he finds the requested name, he switches the program to text entering mode, where he enters characters in braille with vibrating feedback. After escaping from text entry, he instructs the program to send an SMS.

In the next video [6] PG demonstrates how he perceives Morse output pressing the Info key of the program. First he finds the time output, second he finds the date, and finally he finds the battery charging level 90 percent, which he says is enough.

Future Goals

PG uses his morse-braille vibrating phone every day. He has a reliable feedback and his phonebook is increasing. He asked to add full phonebook handling possibilities including searching functions. His second priority request was the alarm clock.

In this final video [8] PG asks us to add a possibility to our program to send longer messages to his computer to be able to read them in braille. We can do that, but we warned him that he has to pay more to the mobile operator for the Internet traffic. We have to thank our deaf-blind interpreter Ildikó for her work in this project.

Conclusion

Dual Tactile Morse-Braille output is a very low cost and usable method for writing and reading SMS messages by a deaf-blind person. He could learn Morse characters and words in a relatively short period of time (3 months). Teaching Morse must be treated as teaching a language. Teaching the recognition of whole words is necessary as early as possible.

References

TACMON consortium http://www.tacmon.eu/new/

Markus N, et al Accessibility for the Blind on an Open-Source Mobile Platform, Mobile Slate Talker (MOST) for Android In: Proc ICCHP 2012 Part II pp. 599-606

Arato A, et al Speaking and understanding Morse language, speech technology and autism In: Proc ICCHP 2012 Part II pp. 311-314

Sigurd Stenersen http://www.justlearnmorsecode.com/farnsworth.html

Arato A., Wigner RCP http://www.kfki.hu/~ha4aa/PG/SMS-sending.MOV

Arato A., Wigner RCP http://www.kfki.hu/~ha4aa/PG/Info-in-Morse.MOV

Arato A., Wigner RCP http://www.kfki.hu/~ha4aa/PG/TrainingMorse.MOV

Arato A., Wigner RCP http://www.kfki.hu/~ha4aa/PG/Requests.MOV