Fieldworks Language Explorer (FLEx) Tutorial: Data Collection to Publication

[00:00:00]my name is scott dysart i'm with sil serving in in thailand and this is a demonstration of flex flex is a software tool that sl has developed designed to help linguists collect data analyze data working from all the way from your initial data collection up to production or publication of data in form of dictionaries grammar sketches or for use in other other products so to show you what flex is i'm going to open up my my example tie dictionary lexicon and here is flex so flex is divided into areas on the left and the bottom you can see we have lexicon if you click on text and words you see the rest of the screen changes each area has its own little view and we'll get into each one of these individually but going back into lexicon which is where people spend most of their time and in lexicon you can add words you can edit words so quickly here i'm going to add a word click on create new lexical entry and of course it goes to the screen i'm not sharing and i'm going to add a tie word i'm going to add the thai word for seal as in the mammal and put in a thai english word a gloss this is a noun this is a real basic entry i'm not getting into any scientific bits just adding a new word and so now i have my word seal so linguists will use this to collect words and add words and sort on words all right um so that's lexicon edit i can if i knew more about seals i can add more information in here pronunciation ipa that sort of thing um so going working down with browse so this is just like lexicon edit but it lacks the entry thing over here so you can't it's not intended for you to be doing to edit fields you can edit individual cells if you want to like a spreadsheet but that's not the view a lot of linguists like to have lots of data lots of columns and so the browse view is a way to have lots of columns you can click over here the little thing here and add add columns just to show the information you're after dictionary is what the diction the your data would look like if you were going to publish it just click on it real quick so this if i were to make a simple dictionary right now this is what it would look like i will get into how to create a dictionary later there are a couple ways to add words one is one by one as i did with seal another is using a technique called rapid word collection the idea is you get a workshop and you have a bunch of people gathered around the table who know different things about the different culture about the culture different aspects of life and then you say okay today we're going to talk about daily life right here we're going to talk about things in the kitchen food so you just say okay write down everything you know about food what foods you eat what foods you what what's good to eat what's good not to eat how do you cook things all different kind of words and then you have a person who collects it and then you have someone you can type it into like into into flex and it's designed to go quickly you have a lot of people doing different parts um so this is a whole method for doing um word collection there there's a website rapidwords.net i believe it is where it shows you how to do this um so so flex is a has a way to facilitate that of course in uh as especially in new languages or languages that don't have an orthography or don't have um well-defined spelling things change over time so flex has a way to handle that with what's called bulk edit entries so i with bulk edit entries you can come in and you can make some changes like for example if um if you've been working in a language and you're doing ipa and you're describing it and you realize that you've used the wrong description you've listened to a lot more and you realize that it's more rounded vowel or something to that nature then and you can use the um bulk replace to go in and change your ipa and you can change all the different valves to a different setup so bulk edit this is something um you really need to practice with and kind of know what you're doing before you kind of get into problems in a reversal it is not enough to say you know the this word has this gloss therefore the gloss is if you're doing like a look up like a tight english english dictionary it's not enough to say chicken is guy therefore guy is chicken in that example it is but there are other ways to do it for example like in english like yardbird is an idiom for chicken so if i wanted to have yardbird in my dictionary i could put in my reversal as a way to look it up i can put in yardbird go referential sense come over here and actually i'm going to find you know chicken because i don't know how to spike i don't know how to type tie and set reversal so now i have a a a set up a reversal so we get more into that later if we were talking about how to make a reversal in the index um so the next thing is text in words the idea with text and word is if you're wanting to do discourse analysis you're wanting to find out how to um how a language is used it's one thing to have individual words in isolation but how are they used to together [Music] you can do this there are a couple things you can do with it so you can use the text in word as a source for words like in this example here i've got this word in gloss it's lesson one i've got information about it i pulled it from a website where i found it what i did with it this is the baseline this is the text i want to go over here to analyze you can see that i've already done some that you can see that this is a name mani so it's a proper noun this is is me to to have but what my dictionary or lexicon doesn't have this word yet so i'm gonna come here and i'm going to add a word so create new entry and this word is i believe it is i has eyes and it's going to be a noun and create so now my dictionary has i've analyzed this word in his eyes if i were to go to lexicon and do a sorry go to lexicon edit or browse even and do a search filter for eyes uh come over here and okay and now there's my word that i should have shown you that that word didn't exist before but it exists now so i'm going to go back to text in words and so now i'm ready to do the next one well we have a something unique to southeast asian languages well maybe not until unique to southeast languages but we see a lot is in there are no space word boundaries so what do you do with that so right now flex thinks this is all one word but that's not so i can go into baseline and i happen to know that right here is the word i'm going to put it in a slash and flex knows that that slash is another way of inserting word boundary i'm going to go ahead and do it right here since i'm at it right at it and go back to analyze and so now flex has already has broken these words and you can see that this blue is saying hey i think that these are the same so i can real quick i can either prove that throughout the text like yep that is always to have or i can improve this particular one because you know in some instances there are multiple sentences to a word form but in this instance at least in this um example i'm just going to hit the check mark plus to save some time and so now you see that this one is no longer blue and this one is no longer blue but let's say i'm not sure if that word is always eyes so i'm just going to but i know it's eyes here i'm going to hit the check box so now that's not this blue color and i'm going to come down here and on second thought that is eyes all right so i happen to know that this is i forgot this is manny that's a name that was correct and so now i have this word and and this word to deal with um i'll just do this word so again it's the same i can add an entry create new entry from this text in word and i'm gonna this is a crow so you can see here that i have when i typed the gloss it did a search it's looking for basically spelling like a couple letters you see these these two tie word characters match these two characters as offering suggestions maybe they're the same um but they're not so i'll hit create and now i have that and so then you just work down the text entry by entry by entry and the the larger your lexicon the quicker this goes because more chances are pretty high you've seen the word before and flex knows what to do with it right and another thing you can do with um text in words is for discourse analysis so here's tagging this is something that's not used very often maybe because people don't really know much about it is you can you can mark subject verb and object so here um i'm going to mark this as in this sentence crow i can't pronounce ty anything but it sounds about right so this is subject this is object direct object and one of these means verb but i'm not sure which one it is but then you can just work down the text and that helps people with discourse analysis another thing you can do is further on in the process it's called text chart um i'm bringing this up here so just showing you that um this course analysis starts out with assuming subject verb object which happens to be what tie is so we're going to leave it alone but um as we know languages have object verbs i mean any way you can come up with a language to be to be written or spoken people have come up with it so flex can handle it but you have to configure flex for it so and in my example here i've already worked through this sentence subject verb object so crow in line two subject uh me is verb and then eyes is object and then i can do this sentence which i haven't analyzed this sentence i mean i know that's the verb i know that's the object but i haven't analyzed that word so i'm gonna leave this one alone so you but you can work it all the way down and through this you can set up all these things to do to help you with your discourse analysis to know how the the language is written right um and of course uh discourse analysis also has concordance so flex actually has two lists of words it has your lexicon that's your text your official list of words with information about the word and it also has your concordance or um words in use how they're used so in concordance um wordless concordance these are all the words that have historically been used in my my project and um if it's ever been in my project it's listed and i can remove them but if it's not being used in the in any text in words now it's got a zero so i can show again this works like the text in word and you can filter you can use edit so greater than one these are all the words that currently are in use in my text and words and it shows you where the word is found um it shows you the analysis that's been done um so you can work down you can see so this is my word for this is called edward uh wants al so you again you can see where it's used how it's used statistics again this shows you told me total number of unique words my lexicon from texting word is not very big but not at all people can get into the thousands in here of course you can do bulk edit like you can in and texting words so that you have to be careful with this because this is like if you have again if you have spelling that's not yet been um formalized or generally accepted but you're collecting text and then if you find out a word is actually we've approved how we're going to spell this word or approved what vowels are going to be implied not written you know making it more efficient once that is kind of set or set you can come in and you can do bulk edit and that will change how that word appears throughout all of your corpus so it's very powerful but like all things very powerful very dangerous so use it with care so that is text and words the way it's interesting to know that text and word and lexicon can be linked so if i were to go to if i go to back up to a lexicon and i'm going to do a search for filter for chicken in my gloss i want to see if i can find where that is used in a in a texting word like if i have a texting word corpus that's big enough i should be able to find my words because again having your words in isolation is fine and dandy but how are they being used that way you can figure out the what the layers are phonologically how it's changed those sorts of things all right here i want to find an example sentence for chicken so i can click on the down arrow next to example and then find example sentence and because i have marked up the word chicken in an entry in text in words the flex will find it so i want to insert this i'm going to hit the little check box here and click add so now you can see example my example sentence here and you can see i have a reference so i can click on example 1.1 and it'll bring me to my example sentence so another thing that is not often used but i find interesting is grammar this is where you can record how how the language is structured um and the various rules and features um the idea with this is if you fill flex in a fill grammar section in as you go the phonemes features rules that sort of thing eventually you can generate a flex will help you generate a grammar sketch which is useful and helpful and then another feature is the is the notebook a notebook is intended to be a again the idea is you you want to contain everything you know about the culture and language in one place and if it works right it all links together so way back when i took uh these cl uh field data methods and management the idea was you as the linguists were always trucking out there you always had your notebook and this is where i show my age of my children say your handy dandy notebook and you're writing down everything you you see in all your observations with careful notations of where you are what village who you're talking to and all that and then you come back and you put it in the computer um flex the notebook is designed to be the place where you put things in about your notebook um so here is an example like way back in 2018 on march 14th i was observing uh some rice farmers at work and so i took notes and i was watching them and i um just then i went back to flex and i started typing in everything i saw and and what i thought and what was going on and and all the little details and so that's the notebook um what is also interesting is that notebook and text and words can be linked together so if you are having a a conver a conversation at the market and you are writing down or recording a conversation with someone at the market you have even if you transcribed the conversation at the market what you have is the conversation you don't have the information or the metadata around that confirmation conversation you don't know necessarily or well you know it in your head but you have to forget eventually where you were at the market is it a you know the things about the conversation so the notebook can actually record the metadata about that conversation that you had in the market and then you can put in the text and so texts and words can be linked to the to the the market so so this assuming i've done it right so this is my in this example this is my the metadata about the conversation and then i can click here and go to show in inch linear text and now i've got this conversation i've got this conversation about mechanized rice farming as a baseline text in words that i can analyze in um in my in text in word and then also i can do it right if i'm in info i can go down here and i can click if i'm if i'm in here and say oh that's interesting i want to show more i want to remember more about that or look into it more i can click here a notebook record and click show record in notebook and it brings me to this and then so if i've got it linked right and if the example i had i was hoping to show had worked correctly i could then have my word in my lexicon i can say oh this is where it shows in a sentence that's good click on that oh tell me i want to remember more about this sentence and if i've got a notebook record that that sentence was captured in i can click on it and i can i can now remember more about the context of that sentence so that's how it's all designed to work together another thing that fieldworks has is lists anybody anybody who's ever worked with data over long periods of time knows that it's important to keep your data normalized it's important to know have the same way of spelling a thing to have the same way of classification so flex comes with out of the box with a series of lists and you can add more lists along the way um so so if i'm in uh my notebook and i have a i'm always going to the same place there's [Music] places or locations so right now i have home so i'm going to add a location i'm going to call it rice field by beside my house or or market or the market downtown or the the the what the big what the little whatever different locations so then you can go back into uh notebook and um locations i can pick from locations and of course it shows up over here i can check box this took place at home and also the rice field by my house and then also participants who are my participants so if i've got you know this thai teacher this person that person and i'm always talking to them i can kind of encapsulate who i'm talking and if i use with and by using lists it greatly increases the likelihood that i will always spell the name correctly or um or if i have reasons to change things this should enable me to change things i could edit the people in list um and i can go to my name and say you know take out say there's a reason to anonymize the last name so if i go down to notebook now all that's changed right so if there's a reason to do that then it can be done so that's um that's a really brief overview of flex itself we've looked at lexicon how you can add words how you can sort words how you can filter words so so that's how you get data how you analyze data the question then becomes what do you do with the data it's one thing to to have your your dictionary become the greatest collection of information about this language on the entire planet but if you don't share it with anybody then then the begs the question of what's the point so flex has as its heart the idea that you're going to want to produce a dictionary and there are well there are a couple ways to do that one is with a print so you can go to go dictionary tools configure dictionary and this is where i can i can change things i can add things that can hide things so just for an example it's a silly example i could i could hide the gloss add the gloss those sorts of things or you can format the way things are appeared with periods and italics style based stuff so you can get hours into this so that's something we can look at much later okay so you do that you you make your dictionary then an output option is pdf and this is an example of a a trilingual dictionary in the philippine language it's about a sambali tagalog and english so we can you can make a pdf that you can then give to a printer to print to a a print book that's one thing or you can put it on a phone or you could do what's uh and if you if you can if you have access to it um sil has a product called webinary this is a a web-based dictionary web dictionary webinary and this is dialer an example of a southeast asian language um and so this is this i kind of view the webinary as like a cultural portal like if you want people to know about daily as an example send them here and so there's some culture information here you can dilute the english you can browse the the this is so this is daila to english um you can you can add links so if you wanted to download uh let's not it doesn't have a download here um you can search you can add some links so you can have a webinary and people in sil this is fairly easy to set up and then you can push to so then once you get entries in flex you can then push up to uh webinary and then in a little while your your new entry is added and webinary is nice is because if you get like a 2 000 words you have it you can push it and people can use it and then you can get 2 000 when you get 2100 you push it up now you have 2100 words in your webinary database and against for people to use um another thing and unfortunately i couldn't get it to where i could share it properly today but um sil has another product called a dictionary app builder and unfortunately i couldn't get it to where i could show my phone on the screen on my computer but i can show screenshots so i'm going to bring this over here and this if you can see it is a screenshot of my phone again this is english symbolic dictionary and you can see this is the right now english i could could have clicked symbol and you'd see some ballet first and then this this again comes out of flex with our dictionary app builder so you can and then it can be made available in the google play store so you make the app you put the app up in the google play store and then people can download it they can have it and then let's say that your 3000 word dictionary now is a 4 000 word dictionary well you then just update the app in dictionary app builder then update the put the new app the updated app in the google play store and then within the process of updating everybody who has downloaded your dictionary gets the new the new up version of the app and so now you have a 4000 word dictionary on all the phones and this is great because good luck adding another thousand words to a print dictionary that's sitting on someone's shelf i don't know where it could be done but with this if they get it off the google play store then you can add it so that's a that's something that's exciting to me and then another thing that is really new as in like very very new is what's called alpha tiles and alpha tiles is an attempt at come on i'll get back here ah okay alpha tiles is an attempt to gamify um spelling and multilingual education especially for minority languages so alpha tiles is a product developed by an sl member in mexico it was gaining wide population popularity over here the idea is you you give alpha tiles a set of words some spelling options good orthography and then you can make a game and the kids can say now how do you spell this word and then you push the buttons and what comes next that sort of thing and it's a game and then you can see as the the checks go across there's some success and built in and the kids like it and um a few as i was telling some people like during the covid season in this part of the world uh when parents go to work kids are either at home on or with them at work with their phone and so we could come up with a game that they can play that helps them that makes it exciting learning how to use their learning how to read and write their language and so that's alpha tiles that requires a set of words and we can get that from flex so those are in brief some ways that you can produce dictionaries or products that use dictionary that use letters in your words and now and from flex and another thing let's go back to my flex is as mentioned earlier and i don't have a good example for it is a grammar sketch when you're wanting to publish what you know about the language assuming that you have filled in the information in flex you can click the generate button and it will come up with a rudimentary grammar sketch that you can then take and edit and then publish so that's kind of nice so this is that's those are some things that you can use flex for another thing being that you can use flex for is if you're in a translation project flex and paratex can link so i'm going to switch over to a new flex project that i have linked to an example translation project all right okay and i'm going to go to an example paratext project so i'm not going to show how you can do it but it can be done so this is an example my example paratex project it's in english this is my example flex project it's also in english so what i can do is if you have it linked you can have flex or sorry paratext come on find word and dictionary in this instance flex so i wanted to find the word daddy so now it's gone through my dictionary and it's a dictionary of three so it didn't take very long but it found the word daddy that it's a noun and it has as a gloss father i can open it in x in the lexicon and i can edit the lexicon it would open up flex if i wanted to do that another thing you can do with this is right click find related words uh yeah that's not okay let's do this find related words to daddy okay so if you have marked up your words with what are known as semantic domains that's when we talked about the rapid word collection those categories of things if you've marked that up and then you're doing a translation you can have peritext ask flex to look for words similar to this with similar semantic domain markings so in my lexicon i've got two words daddy and papa and you can see semantic domains four one nine one two that's father mother king or ken parental kins and also daddy has the same thing so if you've well if you've well marked your dictionary with semantic domains then you can get flex to look up words um another thing you can do um which you can click find a dictionary like i don't have brother i can actually create the word brother if i want to in my lexicon but that is another way you can use your lexicon so i'm going to get out of this so so so far we've talked about how you can use flex like a broad overview we've talked about what you can do with the data that comes from flex but what about if you want to share your data well there are a couple ways you can do that share meaning collaboration not shares and this is a cool word let me show it to you minimize this go to lexicon okay flex uses a thing called curio you don't need to worry about what that is except for that it facilitates the way to to record changes so if i were to click on send receive send receive with other flex users i have a couple options send receive it borrows from your email days of old where you had outlook or thunderbird before you had imap or something or mappy for for gmail you would click send send receive and it would send your emails and then it would come pull your new emails down and it would favor sending over receiving so send would come first then receiving um so that's where send receive comes from it's just borrowing a term all right so unfortunately i didn't think of bringing a thumb drive but in flex there are three real ways you can do send receive to share in a collaborative way so if i were to sharing with you um i could put a thumb drive in my computer and the flex would detect the thumb drive click send receive the flash drive and it would put the data on the flash drive and then i would give you the flash drive and then you do fi send receive get project from colleague especially if it's the first time then you it would look for that you would click on it and it would look for the thumb drive and then bring your data then you can make changes then do send receive to the thumb drive give me the thumb drive back and forth um sneakernet is another way of referring to that so that's offline via thumb drive that's one way to do it another way to do it is with um through a server and right now sal has a service called language depot which is going to undergo substantial change in the not too distant future it just but the the the function i'm not going to show it because how it works will change but the idea behind it will be the same i'll have there's a central server i hook my project to i click internet and do send receive and up it goes to the server and then it mixes my changes with what's on the server and then it pulls it back down and then the next person you if you're on the project would do send receive to the server it would um pull down my changes mix with your changes push um your changes up so it's really even though we call it send receive it's really receive send it's just kind of the way it works um and then it goes back and forth and this works really well and you have a lot of people on the project um to to be working and we have a lot of people doing it another product that we have is language forge i'm going to bring it over i don't have it staged and i don't have a good way to demonstrate it right now but i'll just show what it is language forge is a way for you to share with someone who doesn't really need to be bombarded with all that extra linguistic analysis stuff they just know their language and how to they know words about their language and you can set up language forge to work with your project to where they just get to the computer and they just type words um they type definitions they say they're they're bilingual they know their language and they know a language of water communication they can type their language and then they can type the other language and then that can get meshed in with your flex project and then you can do the analysis in flex um this is kind of by invitation only so you have to invite people into your project so that's language that's another way of collaborating working with other people um go back over here so that is sharing another thing to do um so we we showed how to when it comes to data importation so now we're going to talk about how flex can interact with other projects so we saw that flex peritects can look at flex we're going to show how you can do some other things with other tools that interact with flex i'm going to minimize that so let's go back to flex do my flex project where'd it go minimize this okay that's in the way all right so a tool we have the use of is something called speech analyzer and speech analyzer is a tool that can again take several hours of demonstration we're not going to get into how speech analyzer works except that you can take a wav file of someone speaking and you can mark their uh you can isolate the words like this is a word and you can see tone and things and then if you do it right you can chop this file up this wav file in such a way and mark it that now i have i actually have several words here then i can export this in such a way that i can now import it into flex and this is really fantastic in that by so reference non-blanks so these are the words i don't yet have tie for these so these are the words that i just imported into speech analyzer you can see the reference five six seven eight nine and so here's my the ipa for it what's great is the source for this word was not me typing the source for this word was an audio file and when you export it out of speech analyzer speech analyzer will chop that file that wave that large wave file with that large sentence or whatever it is list of words and the little bits that surround the reference so now if i do it right you can click the play button and you can hear the word that actually didn't uh that actually didn't click right but the idea is you can get the word and then so all these now have audio files mwah that's more better actually missed that's too funny again the dangers of live demonstrations but that's the general idea another way that you can import so that's importing individual words but what about if you're you have collections of sentences that you want to to work with let's say sil has another tool called say more sameore is a tool that will you can use to record people and transcribe their session and you can annotate it and then you can export that into what's called a flex text file and then you can import it into text in words so if i go to text in words i could then have text in words another thing you do you can do real quick before we run out of time here is in phonology assistant it has a way of taking your ipa so in my text in words my lexicon i've marked up a lot of ipa so in phonology assistance i can then do analysis if i want to find search for all words let's start with gur grow so these are the words that grow up like moo grow up which is a lovely way to cook pork um i can i can find those and then if i realize that and if i have got it sound set right i can go to data corpus find pi crispy which word uh if i've got an audio file for it i can play it red toaster has a word of course it's not playing right now naturally naturally is not playing right now but if i keep so you can either you can hit f5 to play it or ctrl f5 to play it over and over and over again as you're listening to it and you say and if you realize that you want to change the way it's edit change the way it is pronounced or spelled phonetically you can click edit source record and it will bring it into bring it up into if you had flex close it would open up flex and it would bring it up so that is a very fast hour on an introduction into flex