Solo GIS Consulting with FOSS4G: Workflows and Tools

[00:00:02]testing 1 2 3 brew on and get started on time because it's a pretty full presentation I'd like to leave some time for questions I'm Kurt Menke or if you're on Twitter you may know me as Geo mankey and the title of my talk is adventures of a solo GIS consultant using GIS and foss4g so I'm based out of Albuquerque New Mexico and I used to be an archaeologist this is actually my second career so I kind of picked in Indiana Jones theme for this talk two things I love maps and archaeology so the funny thing about archaeology is that this is what it felt like Indiana Jones in the map room and this is what it really looked like just a bunch of you know dirty people digging really square holes in the desert you know so that's a much younger me in the lower left-hand corner and the blue tank top shirt eventually I mean that was really fun for my 20s and I loved archaeology but I decided I needed a real job eventually got a master's in GIS and have been doing that ever since so I now call myself a reformed archaeologist and I've been doing j/s like I said for about 20 years at this point and I've heard a lot of people say this week that this is their first foss4g raise your hands if this is your first foss4g event so quite a few of you I think that's a really special moment I put together this map of foss4g that I've been to and with all the little logos and I've been to almost all of them on this continent since it started and the first kind of foss4g ish event that I know of is the map server meeting in 2003 and there wasn't even a logo for that so I just put the picture of the group photo from that 2003 meeting up and this was a really small intimate meeting about 125 people I think there's the official tally for this and it almost felt subversive to me at the time because you know breaking away from ESRI at that in 2003 like a really big deal got to there are some great workshops by Steve Lyme of mapserver and Tyler Mitchell for post Jess and Frank warmer damn for doodle and ogre it was great and we were kind of wondering where the ESRI spies were collectively at this thing and that's and another picture of a much younger me there in the in the back row at that meeting so after that I was hooked I've been coming to all of these pretty much ever since you know the next year we had a nice one in Ottawa and I think the first actual foss4g might have been in 2006 and lose on something like that I wasn't able to go to that but I went to the 2000 meeting in Victoria which was amazing and they've just continued to grow and you know getting bigger and bigger and bigger and this year's another example of a record-breaking phosphor GNA I've tried to go to most of the phosphor GNA s as well and last year I decided to go to the first hugest developer and user meeting in Denmark so I have this little inset in the upper right for that meeting in Denmark and this is the photo from that and this felt to me a lot like that very first map server meeting it was small and you know really super friendly got to meet the whole brain trust for QGIS it was amazing I mean Anita grazer and Tim Sutton and Niall Dawson and Mattias Kuhn all these people that I've been seeing on listservs and blogs for years I got to meet face to face everyone was super friendly everyone spoke English really well so that wasn't a problem and I I would say if you ever have a chance to do something like that do it it was it was a fabulous meeting and that's a picture that's me and a little red circle in the back of that meeting as well and the other cool thing I was the only America at this meeting and in fact there was only two people from North America at it so it was just a really cool experience so a little bit about what I do so I spent the first 11 years of my GIS career working at a place called the earth data analysis Center at the University of New Mexico that's where I kind of cut my teeth with GIS and eventually it got to the point where I was you know a primary producer doing all the work eventually they started wanting me to manage the budgets for the project I was working on and eventually he wanted me to start working on proposals and at the same time the work was topically getting less interesting to me so I started thinking well what do I need you for if I'm doing everything from proposal to final product surely if I apply myself I can make this work so these days my two topic areas are mostly conservation and public health GIS consultancy is the bread and butter of how I make a living I was it was a progression I started my own business in 2000 when I was still working at the University and there were eight years there where I was kind of managing a gradually growing side business on the weekends and working full-time when I really decided I wanted to make the jump I was probably in 2007 working two full-time jobs and thank God I was a little younger because I could handle it most of my conservation clients are nonprofit conservation organizations that are kind of watchdog groups for the BLM and the Forest Service and it agencies like that so I wear a lot of hats too I've recently become an author last year I authored mastering qgis and just this week I put out a new book called discover Kiedis that I'll talk a little bit more about I'm also involved in education so I teach part-time 2008 when I went solo is when the economy hit so it was a little scary and so I started teaching just to diversify a little bit to have multiple streams of income and discovered that I loved it so I've been teaching ever since I can't afford to do a lot of teaching because it frankly just doesn't pay much part-time instructors but I just do it because it's a way to give back community and I really like it I've also been involved in curriculum development I'm a geo for all member and last year I was honored to be nominated as an AWS geo charter member and I'm also being paid to be a blogger these days so a lot of different hats one of the main reasons I really chose to work for myself was autonomy and self-determination I really just wanted to have the freedom to work on things that were important to me and work with software I wanted to work with and you know work with people that I liked and all of that because life's too short to be working on things that you don't enjoy so I've kind of had a intermittent ongoing migration to foss4g my first introduction was in the early 2000s when I was working at the University and we were standing up a lot of our chi ms web mapping sites and we started having this problem where we come in every Monday morning all the map services were dead we'd have to relaunch them all and it was just a nightmare it was not working you can't have a website go down like that so we discovered map server and started working with map server and not only was it faster but the stability was crazy we would stand up these sites and they would just run like clockwork for years without us even really having to do any maintenance on them and we were doing a lot of so I spent from 2001 to 2008 my main job at the University it was mostly web map development with map server and sometimes grass as a back-end and things like that somewhere in there on 2006 I discovered QGIS it was about version 0.7 I think and it was just a glorified data viewer at the time and but I was interested in it and so I kind of followed it through the years I would always install the new version to see what was happening this dotted line is kind of the period where I went solo and so in 2010 I began teaching a full semester class on intro to open source GIS and web last year I 2014 I developed the Geo Academy with some colleagues and that led to co-authoring mastering hugest and these days hugest has finally gotten to the point I should preface it whereas I used to do a lot of web mapping these days just because of the what's asked of me I'm doing more spatial analysis and desktop GIS than web mapping and so I use a lot of QGIS and that's what I'll talk about mostly in this talk in terms of technology and grass and Google and ogre and Postgres and all of those kind of desktop tools and I'm using those for almost every new project that I take on at this point so some of my reasons for using foss4g I think you know the most important one for me really almost comes down to the community and these kind of range from practical reasons to sum that up just kind of support my mission and values but the community of people working with foss4g is tremendous and that's why I like coming to these meetings because I'm always meeting new people and I'm frankly in awe of all the people that come to these four G's and all the work that people do because there are some brilliant people coming to these meetings that it's very humbling and so that great communities also provides great support there's really nice access to developers I mean I can't talk to Ezra developers but I can talk to you know Niall Dawson about some new feature of QGIS and get some really useful feedback you know and then most importantly the software works it's interoperable it's fast so this is a photo from that Denmark meeting some of you may know Jeff McKenna who is the past president of oz geo lean Fischer's in the middle there who is a professor in Copenhagen who organized that meeting and then Tim Sutton the head of the hugest PRC so it's just so cool to be able to just you know rub shoulder to shoulder with these people so to go through some of the practical aspects of QGIS especially that I'm a big fan of there's all the database and OGC connections so you can bring almost every kind of data you'd want in theirs at this point I think 42 different vector formats that QGIS will read 88 raster formats so there's almost there's really nothing that I can't bring in and work with and cue just some of my favorite cue just plugins there's a lot but these might be some of the the highlights so if you've used data to find values to do label placement in QGIS there's a great plugin called layer 2 labeled layer that creates all those attribute columns automatically for you status and group stats I use a lot for creating summary tables and getting stats out of the tables I do a lot of reclassification of raster's and so there are some grass tools for that but there's a really nice plug-in called slicer that's really handy for that as well quick map services for base maps mmm QGIS has a whole suite of tools but the ones I most fond of are the geocoding tools in mmm QGIS because you can batch geo code a bunch of data and then cue just the three Jas is a fantastic plug-in for rendering data 3d in a browser then there's all the cue just geoprocessing tools there's the all the different providers in the toolbox you can work with models in the graphical modeler you can batch process with all those tools you can batch process model tools and then there's also obviously the the Python console and one of the new features of key is that I'm becoming a big fan of our project variables have anyone in here even if you've used project variables yet no one this is a really powerful new feature that just came out with 2:12 I believe and you can set them at the layer level or the global level but one thing I have to do a lot is calculate acreages and square mileages of polygons so I've simply set up global variables for the conversion factors between square meters and acres square feet and acres and now when I open up the field calculator I can just quick we calculate those numbers using variables so I make about two to three hundred Maps a year I don't really know how that happens but I've tabulated it recently and that's what it always seems to come out to and I'm a really big fan of the cartographic capabilities in QGIS and I get to work or some pretty interesting data because of the organizations I'm working with this is a map from 2011 that shows the Powder River Basin in orange and northern Wyoming and southern Montana and then it shows all the coal-fired powerplants that coal from that field gets shipped to and all the ports as well I also love the live layer effects and inverted polygon shape burst fills so I have clients that unfortunately want the kitchen sink on a map that can be really challenging they're always wanting multiple polygon layers on top of one another I'm always explaining you can't really do that very effectively and maybe they need two or three Maps instead but with QGIS these days I think some of the cartographic effects are really powerful this is a map of the the Gila bio region in southwest New Mexico where the Mexican wolf population is trying to be restored and it's a pretty rocky program it's not going so well and one of the issues is year-round livestock craving grazing and calving operations and so I'm working with an organization that is working with these grazing allotment permittees to retire those allotments so that there's just fewer cows and fewer chances of depredation out there and so they wanted to highlight a couple of these allotments in that are they're working on getting retired on this map the orange polygon is the home range for wolves in the most recent year and so you know it's a busy map but the inverted shape versus polygon filling the bio region caused it to pop off the map a little bit then I was able to use some kind of subtle drop shadows on the small allotments and the wolf home range layer that just make it a little easier to get the point on the map and one of the interesting things I found is that my clients they don't care what I use to make these maps some of the clients I've been working with for 10 years and I know pretty well and I've explained open source or tried to a few times and it's really just too much information for them they jet they they kind of think it's neat they don't really care what's being used so it's just for me really this is another interesting map that I made a few years ago there's a federal agency called wildlife services that traps varmints around the western US and their data had been locked in filing cabinets for years and we fired all that data and had some interns enter all of it by county and then so we could release it onto a map and show this map that most people don't see very often which is shading the counties by how many animals have been trapped and killed by this wildlife services department the losing County was Elko Nevada with almost 18,000 wildlife trapping in in that 10 year period so one thing I don't you know I'm always working in usually roadless areas or you know in areas like that where OpenStreetMap data isn't so useful to me and I feel like a little bit of an outlier because open story up is really cool but it doesn't usually apply to what I'm I'm doing so another aspect of my work is modeling wildlife habitat connectivity or perhaps you've heard them called wildlife corridors and there's a really powerful open-source tool called circuit scape developed by Brad McCray at the Nature Conservancy that's used to model connectivity between habitat areas with an electrical paradigm basically a circuit theory then there's now a QGIS plug-in for circuit scape that allows you to run circuit escape through the processing toolbox which is really nice and this really busy psychedelic map kind of shows the output of a model like Maran a couple years ago using circuit scape which is this is riparian habitat connect and it's species generic it's just looking at basically the connectivity of you know taking in surface water and so the the brighter areas would have the highest conductance through riparian areas and the darker purple areas would have the most resistance and I'm gonna have to go through most of these pretty fast I could I've actually given like full presentations on some of these projects solo so I'm just kind of blasting through to give you a taste of some of the things I work on this was a model I did for this cute little bird the Ben deers Thrasher which is its populations are just crashing in the southwestern US this is just a plot that shows the population decline in the last several decades and no one is really sure what's driving this but they're worried that the bird to be completely exterminated from New Mexico in the next couple years if nothing is done and so New Mexico Game and Fish hired me to model habitat for this bird and there's a long story behind this but I use breeding bird survey routes to as the occurrence points I used a inductive niche model to generate habitat for this so it was based off occurrence points and these little red triangles are the occurrence points I was able to mine from the breeding bird survey data which was by far in a way the hardest part of this project and for this I use the software called MaxEnt it's a maximum entropy model for computing niche models for species habitats it's not open source but it's a freeware package developed by Princeton that you can download and it's a Java based GUI and it basically lets you take a bunch of occurence points and on the right you can feed it a bunch of environmental variables that you feel are important towards determining habitat for that species and it's it's been demonstrated to work really well for species with very restricted ranges and for species you don't have a lot of occurrence points for so it was a really good fit for this bird and you can use categorical or continuous raster so some of the inputs are climate based like average annual precipitation some of them are vegetation inputs some of them are elevation slope those kind of things and this is the resulting model output where just the the small green and yellow areas would be the most optimum habitat for the bird and the red would be essentially non habitat and the really cool thing about MaxEnt is it once it has a bunch of tools for indicating the predictive power of the model and the importance of variable contribution to the model but it'll also let you project that model onto a set of future climate change data once the model has been built so I did that with some data from prism from 2050 which resulted in this which we were shocked at because the habitats we thought was shrinking so I think there's a couple things here one there's probably more the bird is using to identify habitats with then I can represent in a GIS - there's some other species that are probably out competing this bird so we actually did a model for a curve bill Thrasher to show how that was interacting with this habitat so another aspect of my work is teaching I really see teaching as a bridge this is another Indiana Jones metaphor here between the developers and the end-users because if people aren't teaching foss4g then people don't know about it and no one's using it so I've been teaching foss4g since let's see 2009 was the first time I taught us full semester foss4g open-source class and its really Disick sighting it's it's fun to turn the students on to this they're really excited when they realize I can install it on a Mac first of all and that it's free and why is no one talking about this so I mean kind of the usual introductory questions I get so that's a lot of fun and I've also as part of that been involved in curriculum development I've developed a full semester class and then in 2014 dr. Phil Davis pictured here this is kind of his more normal habitat near a sailboat that's where he prefers to be he was our fearless leader for developing this thing we call the geo Academy which is a set of five College courses to teach GIS using open source software so it's more teaching GIS than QGIS or some specific software but it's using huges to teach GIS and it's all out in the Creative Commons and so we were kind of thinking like we build it they will come teachers can use this and take this curriculum because most people don't have time to develop all this so we were basically developing this and giving it to people for free it's based on the Department of Labor's gtcm which is a hierarchical tiered model of the skills knowledge and abilities you have to have as a working GIS professional today in today's market and so it's been vetted by dozens of college professors and we were honored to win the geo for all international educator team award last year for developing this curriculum and I've also developed curricula for a couple other initiatives so that's a lot of fun and the exciting and very unexpected result of the Geo Academy for me was the invitation from Packt publishing to author mastering huges so our whole team co-authored this book last year which is a lot of fun and I'll tell you it's true that when you teach you learn as much as the students do it is also true if you really want to learn software write a book about it because you know we all kind of have our little pathway through a workflow but you know you're not paying attention to every little single checkbox on that GUI and when you write a book you have to investigate what each one of those things does and so I learned a tremendous amount even though I thought I already knew a lot by writing this book and another benefit of going to that meeting in Denmark was I got to meet one of my co-authors dr. Luigi Pirelli he now works for Bao but he's one of the core cutest developers as well so we were also invited to update this book this year because Q just developed so quickly we're now pretty much done with the second edition of this I'm trying to figure out from pact one this will be ready it's in final production and so we gave it this nice snazzy cover of this is Essen in OpenStreetMap beta which is the name of the latest release and it should be due out any day and I've been saying that for a while Rudin reality might not be out for a month or six weeks but look for that soon it's updated to the latest version of QGIS and it has some new chapters of additional material another shameless plug so I was also invited by locate press which is run by Gary Sherman the inventor of QGIS to publish the Geo Academy curriculum as a workbook so I do worked over the winter on that it was basically just a matter of formatting it into a workbook and I also updated it so it's now the most up-to-date version of the Geo Academy curriculum it works with hugest to14 grass 7.03 and Inkscape are all components of this curriculum so this is all five college classes all as a workbook with each one has challenge exercises study questions and things like that and right now if you go on to locate press it's available as a preview Edition because we still have some formatting tweaks to make to the book but you can download it as a PDF the list price is going to be $36 right now with the foss4g and a coupon code you get a 50% discount and you'll be eligible for the final release when it's ready in a few weeks so another aspect of my work is working on this contract with the National Library of Medicine where we're working to empower public health groups and communities working with underserved populations and we have a blog as part of that community health maps NLM NIH gov and there are some resources there and we've worked on case studies and so we've worked on quite a few for example where we've been to Hawaii and worked with Native Hawaiians and help them map their native knowledge of winds and rains and some really cool that worked with obesity data out there we've been to Seattle and showed some people up there how to map noise pollution and in Indian neighborhoods all mapping their neighborhoods with their smartphones and using QGIS to make maps and cartodb as well and then Charleston South Carolina most recently we've been down there several times and taught them how to do the same thing and one of the highlights was going to this high school on one of the sea islands outside of Charleston and showing these women over their lunch break how to use fulcrum in their phones to collect data and they picked it up like that it was a lot of fun and then they spent the rest of the semester mapping their island and so it's really exciting to turn people on to this technology that wouldn't normally know about it or have access to it and so some of the challenges of switching to foss4g this is what it feels like sometimes to me first of all I have thousands of MX DS on my I've been doing GIS for so long but I have just a lot of work that is in MX the format that would be above an MX d2q just converter you know I often have clients that require an ESRI geo database or an ESRI model as a deliverable which I can do so this is an example of one of the models I'm talking about I'm working with the Wilderness Society right now on this massive ESRI model this is actually a model that has two dozen sub models in model builder that took a couple people a year to create and it takes a National Forest Road data set bangs it up against a whole suite of invar minimal variables takes a couple hours to run on a national forest data set but identifies routes that could be candidates for decommissioning and then it ranks those according to those environmental variables and it's just not worth rebuilding the wheel so you know one of the things you have to realize is you know when to use the right tool France and there are very very rare functionality issues one of my issues right now that I've told some people about is support for raster attribute tables there's some categorical raster data that I get as such as land cover or vegetation data that have multiple attribute columns and I can't access that in QGIS or grass I don't think I found out that maybe I can work with it in R or Postgres and so I need to look into that but I've submitted a feature request to queue just for this and so hopefully one day that gets answered so go to help resources I think one of the coolest things now is it this is my one part of my bookshelf there are so many books out there now and I want to thank Gretchen Peterson and Anita grazer for making cartography cool again with their new QGIS map design it seems like cartography always kind of was kind of poo-pooed and now everyone's really into it I think that's cool so I also you know follow GIS tribe on twitter twitter meetup every wednesday there are which is a lot of fun I use Twitter a lot to keep up with latest developments I go to blogs like Anita grazers and Niall Dawson to follow QGIS updates and then there's also you know GIS Stack Exchange and of course all the listservs that you can answer questions with and I find people be really helpful so what's next well there's always the challenge of being current and learning new tools and I find this to be very difficult because I don't have a lot of spare time so I might really want to use are but I have to figure out a time when that fits into my schedule so I really need it to be something that I can see a direct application to on a project I have going and I don't always know enough to be able to figure out that application and so I kind of just put it on the back shelf and hope that someday in August I'll be able to get to it or something like that another thing that I'm going to be doing is working with some lawyers on I there's a big forest fire in New Mexico and working with some lawyers who helped an Indian Pueblo see the Forest Service for damages from this fire and I'm gonna be an expert witness and a trial this fall to defend some of the exhibits I created which is going to be a new experience for me I'm also gonna be working on a mountain lion habitat model across all 11 western states which is gonna be a big challenge I know there's a lot of state agencies wanting to migrate to open source technology I'd like to assist them in that but I haven't really found the opportunity quite yet and obviously attending more foss4g events it's always high on my radar so some closing thoughts one thing I've been thinking about is I've I've forgotten as much as I've learned there's the technology in the last 20 years I think have switched over at least three separate times in my career like I used to be an awesome Avenue programmer and who cares anymore you know so don't be afraid of new tools you just have to keep current it's part of the exciting part of being involved in G is just that rapid pace I think also GIS is a tool and you have to have a niche I get about a letter a month an email a month asking me someone would like to pick my brain about how they could do what I do and I don't have a formula for it I don't even really know how I've made this business of mine work but I think one thing is having an itch like - conservation another thing is being easy to work with and so I I know that I from people told me that I've gotten I've gotten work because people find me easy to work with and so like for example I have clients that have an internal GIS shop and they find it easier to contract with me and faster turnaround than to work with their own internal people so that's another thing I like and I'm self-employment it's a balancing act like we all we all have there's a lot of things to keep current so you have to be a multitasker but I think there's a lot of more important important work yet to be done so if you're if this sounds exciting to you I encourage you to do it and so that I can take any questions you might have the question was have I customised an interface or customized key just in any way for for clients and I I haven't done that I'm at the at the moment I'm providing resources directly maps and analyses I used to create custom ArcView applications for people I've done that in the past but I'm not doing that currently but it would be fun to do yeah the question was I hire subcontractors and I do often that's one of the things teaching has benefited me you know I always know the rock star students that need work and so I've used students like that several times and collaborated with colleagues occasionally a good question the question is how how do i balance you know the he coming to foss4g and working for clients I don't maybe not have a deep pockets and things like that and I you know honestly I've done some pro bono work but I basically if the client doesn't have money most time I can't work with them but I work for nonprofits if I have a very reduced rate I'm pretty cheap for my experience level I think I have a sliding scale depending on whether it's a for-profit or a nonprofit organization coming to events like this is very difficult for me I put this on a credit card you know I'm going to pay it off so I'm both not making money and have money going out so it's it is very difficult to come to these and I have to pick and choose which ones I go to but you know I've done very well I don't I'm not getting rich but I'm comfortable and so it seems to work the question was was though is it what's more important a niche or I guess using the just the technology and hanging out a shingle or not I think I mean personally I think it's important to have a niche i that up over eight years before I went on my own and I had I knew I had a stable of clients that I could work with and I also had knowledge about the area that I was applying it to so I'm not a biologist by training but I've learned a lot about ecology and conservation over the years because that is what I do and so I'm going to expert in those I know where to get the data quickly and I know some of the issues with mountain lions or National Forest Road datasets and things like that and that is so I'm able to apply my knowledge in that way and so I think if you think that's something that's important to develop is an interest or an area that you're going to apply the tools to well thank you everyone