Using Input/Mergin and QGIS for Field Data Collection

This post will show the basic steps for working with an alternative to Fulcrum for community data collection with a smartphone. The app is named Input. It’s a mobile app developed by Lutra Consulting. The app is free. Until now it was only available on Android. However, Lutra Consulting is working on a version for iOS! The Input workflow is based in QGIS. This means you set up your data layers in QGIS, along with the logic for your data collection. (This will mean a steeper learning curve for those not already familiar with QGIS. But don't let that discourage you, it is not that much steeper!) You then use their Mergin  cloud service to synchronize your QGIS project and data between your desktop and phone. There is a Mergin QGIS plugin making this quite simple. All the symbology you set up with your data in QGIS will be honored by the Input mobile app. With the combination of QGIS and Input/Mergin you can mimic all the field data collection functionality covered in a typical Community Health Maps workshop!

Getting started

Here I will walk through the steps in setting up the same type of data collection in CHM workshops. (Note: this post was inspired by the well done video by Dr. Hans van der Kwast on using Input). To get started:

• Create a folder for your project
• Open QGIS (in this example I’m using v 3.8).
• Install the QuickMapServices plugin and add the OSM Standard and Google Satellite basemaps.
• Zoom in to your study area.

  

This step is optional but helpful. You will set up a layer as the study area boundary.  From the QGIS menu bar choose: Layer | Create layer | New Geopackage Layer.

• Save the Database  (*.gpkg) into your folder.
• Table name = study_area
• Geometry type = Polygon 
• Click OK.

Now that the study area has been created, the polygon for the study area can be digitized and then styled. Right-click on the layer and choose Toggle Editing from the context menu. Use the Add Polygon Feature tool on the Digitizing toolbar to trace your study area. 

Styling the Study Area

Clicked F7 to open the Layer Styling Panel.  Here I gave the polygon a Fill Style of No Brush, a Stroke Color of bright red and a Stroke Width of 1.26. This study area covers a school and park near my office.

Creating the Infrastructure Points Layer

This is more important. Create a second layer into which you will record points out in the field. Repeat the above steps to create a point layer and save it into the same GeoPackage. Name the Table infrastructure and set the Geometry type to point. Before clicking OK you will add columns to record the data you will collect. For each new column, enter the name, set the Type , Maximum length and click the Add to Fields list button. (Although not in the screenshot below you can also choose to add a date column - type = DateTime.) When you have added each field click OK to create the layer. When saving, choose Add New Layer so the layer is added to the existing GeoPackage.Now symbolize the points. Here I have used to Simple markers to create a purple target icon.Save your QGIS project as a QGS file into the same folder. (At the moment Input does not support the zipped default QGZ project file format.)

Creating Offline Basemaps

If you have a cellular connection the basemaps will work fine. However, if you anticipate losing cellular connection out in the field, you can use the Generate XYZ Tiles (MBTiles) processing tool to create offline versions of each basemap. You can set the extent to that of your study area layer. You will need to experiment with the zoom level settings. When they have been created you can use the Browser Panel to add these to your map. 

Working with Map Themes

You can also set up Map Themes. These allow you to have different views of your map. Clicking the eye icon at the top of the Layers Panel will open the Map Themes menu. For example, to set up a map theme for just the Offline Satellite layer the study area and your point layer, you would just turn those three layers on. Then choose Add Theme and name it Offline Satellite from the Map Themes menu. Here I have set up several themes. With these I can easily toggle between these different views while in the field. 

Setting Up Field Widgets for the Points Layer

The next steps are key for mimicking the data form functionality in Fulcrum. You will open the Layer Properties for the point layer, and switch to the Attributes Form tab. Here you can select each Field, and set up custom widgets which will control the editing behavior in the field. Below are the fields typically used in a CHM workshop.

• fid - you don’t need to see this field, so select it and choose a Widget type of Hidden.
• Infrastructure type - Select a Widget type of Value map. Here you can enter a series of choices with 1-x values. Under Constraints check Not Null which makes this required data.

• Bike rack capacity -  Select a Widget type of Text Edit. This allows the data collector to type the number for this. Since this only needs to be answered if the infrastructure type = Bike rack you can enter that as an expression in the Constraints section.

• Num bikes - set this up the same as the bike rack capacity.
• Tree type - Widget type = Text Edit and you can set up the appropriate Constraint expression for this.
• Sign type - Widget type = Text Edit and again set up the appropriate Constraint expression.
• Photograph - Widget type = Attachment. In the Path section click Relative paths. Then scroll down to find the Integrated Document Viewer section. Set the Type to Image.
• Date - Set the Field Format to Date. Uncheck Calendar popup. Under Constraint check Not Null. Then in the Defaults section enter $now which will default to today's date. 

Close Layer Properties and save your project. 

Project Properties

Now there is just one more setting to create. From the Project menu choose Properties and switch to the Data Sources tab.  Here check the box making the study_area Read only. This layer is just for reference and will not be edited in the field.Save your project.You are done with the set up and will now use the Mergin plugin to migrate your data to the server. In QGIS install the Mergin plugin.

Setting Up Mergin

Visit the Mergin web page (https://public.cloudmergin.com) and set up an account. With the free account you get 100Mb of data storage. (You can inquire with Lutra Consulting about getting additional cloud storage space.) In the QGIS Browser panel you will see a new Mergin data provider. Click on it and when prompted enter your credentials.In the Browser panel again right-click on the Mergin icon and choose New Project. Fill in the details and click OK. The data will upload to your cloud account.

Collecting Data with Input

Now that the project is uploaded to the Mergin server you can switch to your smartphone. Install the Input app. Log in with your credentials and you will be able to access your project from the Projects list. You can then use the More option to switch between different map themes.When you are ready to collect a point, click the Record button and then the Add Point button. The form will open allowing you to collect all the data for that feature including a photograph. The points which have a condition preventing them from being entered will be unavailable and red. For example, the animation below shows a Stop sign being collected. The fields related to other things (bike racks and trees) are unavailable because a condition has been set within QGIS. Once the information has been entered simple click Save and move to the next data collection point.

Downloading/Synchronizing the Data

When you are ready to download the data you can either 1) Use the Mergin data provider via the QGIS Browser panel to Synchronize your data, or 2) Synchronize the data via the Mergin website.You can then open the updated project from the same Mergin provider and begin to work with your data!This is still a very new app but shows immense promise for Community Health Mappers. I encourage Android users to try it out. iOS users stay tuned for future developments.

QField - A QGIS Related App for Data Collection

QField is an application for collecting field data via an Android device. It was started 4-5 years ago by the Swiss company OPENGIS.ch LLC which also employs several core QGIS developers. QField has reached the point where it rivals most data collection apps. The only reason we have not been using it for Community Health Mapping workshops is that it is not available for iOS. This is simply because the open source license used by QField does not allow it to be wrapped into a proprietary software license like the one Apple employs for it's store.  If however, you are a Community Health Mapper who uses Android it is a fantastic choice.NOTE: It is possible to set up an app which is compatible with iOS but does not participate in the App Store. This solves the licensing issue. Setting up an app this way necessitates becoming part of the iOS Enterprise Program which costs money.  QField developers would like to make this happen, and it will likely involve iOS users donating to QField.Although QField is rooted in QGIS code it is not a miniature version of QGIS. Rather it is a streamlined data collection app. As they say, "the buttons are few and they are large," so you can work with it out in the field. QField lets you create a map in QGIS and upload that map to your mobile device. From there you can collect data.The workflow for QField looks like this. You begin by making a QGIS project on  your computer. Importantly this project will contain the point, line or polygon layer(s) you want to populate in the field. (NOTE that Fulcrum only allows the collection of points!) This means you think of your survey form and data to be collected in the office, and create fields in your GIS layer(s) for each question you want to answer. With a little bit of QGIS editing familiarity this isn't any more difficult or time consuming than creating a form in Fulcrum.  You then upload the folder containing the QGIS project and data to your mobile device. The GeoPackage data format, which is the default for QGIS 3.x, works great with QField. There is even a QGIS Plugin named QFieldSync that facilitates migrating your project to your device. Once the data has been copied to your mobile device you can open your QGIS map file using QField.QField uses the same rendering engine as QGIS so the map will look identical to how it did in the office.  Once the map is open you can select from one of two modes: Browse or Digitize. When collecting data you would choose Digitize. Then select the layer you want to work with.For public health officials with security concerns QField is a great fit because there is no third party cloud platform involved. The data is not being streamed across the internet. All the data is stored locally on your mobile device. You can simply use the My Files app on your device to navigate to your GeoPackage file and email it to yourself. If it is too large you can connect your device to you computer to download it or use a data sharing app such as DropBox or Google Drive.Since this is an open source project you can request new features and report any bugs you encounter by contacting the developers! Since QField doesn't cost anything to download and use, you can also consider donating to the project to help it continue. Even small donations are helpful to projects like this. Doing this makes QField better for everyone. I encourage you to try it out.

So What is Open Source Exactly?

This term ‘Open Source’ pops up in discussions about Community Health Maps and the blog and I realize many may not really know what it means. That’s OK, I’m going to explain it here. From both licensing and software development perspectives, there are two broad categories of computer software: proprietary and open source. The figure below shows some examples of both types.Proprietary Software Proprietary software is created and sold by a corporation. They create software and sell it to make a profit. When purchasing it you may also be paying for the privilege to get help and support using the software. Or you may have to pay extra for that privilege. Two examples of proprietary software are Microsoft Office and Esri’s ArcGIS. There is also a license that accompanies a proprietary software package. When you purchase software, you are actually just buying a license that gives you the rights to operate the software. You never actually own the software itself. That software license will restrict use in some way:

1. the number of computers you are allowed to install the software onto,
2. the time period that the software will operate, and
3. the number of features you are licensed to use.

Open Source SoftwareOpen source software (OSS), on the other hand, is created by a community of software developers (programmers). It is created to solve a common problem and is made available freely for everyone’s use. Open Office, Android and QGIS are examples. Open source software also comes with a license. That license tends to grant rights to users. For example:

1. The freedom to run the software for any purpose.
2. The freedom to study how the software works.
3. The freedom to redistribute copies so you can help your neighbor.
4. The freedom to improve the program.

Access to the source code may not be important to most, but the freedom to use the software as you see fit very well may be.Why Give it Away for Free?To many it is counter-intuitive for something of value to be given away for free.  However, open source software development isn’t entirely altruistic. Most of these programmers work for companies providing services with the software. Those involved in an open source project simply feel that this is a better way to create software. There are several reasons for this:

1. there are a lot of “eyes” on the code and bugs can be spotted and fixed quickly,
2. having access to the source code let's you understand how the software is working, it's not a 'black box,'
3. you are not locked into a particular vendors system, licensing scheme or software release schedule,
4. you have the freedom to create any missing functionality that you need
5. you benefit from the contributions of others and vice versa.

Making Money with OSSObviously you can provide services using OSS. However, there are other ways to make money with open source. Companies like Facebook, IBM, Sun, and Google are all heavily involved in open source. Let’s take the example of Google. They may be the world’s largest open source company. One of the keys to their business are vast server farms, which include several million servers. Google never would have been able to get their company off the ground if it weren’t for the Linux operating system. The cost of putting a proprietary operating system like Windows on all those servers would have been prohibitive. Now their open source Android operating system is the most popular on the market with a 66.5% market share. In another example, even Esri's ArcGIS includes some open source software components 'under the hood.' This is because some open source software licenses allow that software to be bundled with proprietary software and sold for profit.Any given open source software is considered a 'project' and they aren’t all user friendly and useful. You still have to determine if the software will meet your needs. An important aspect is getting help and support. Things to look for include a good online manual, ‘how-to’ books, and an active listserv.QGISLet’s take QGIS as an example. It has fantastic support. It has a great online manual, training material, case studies, sample maps, commercial support, email listservs, plus a number of 'how-to' books written by people who use QGIS. The QGIS project does not have a corporation behind it. It has about 30 independent but dedicated core developers. They work in a democratic fashion, voting on new features to be implemented. As an end user you can provide input with feature requests! With QGIS, if there is a feature you need that doesn’t exist, you can hire (sponsor) a programmer to create it. It then may become part of the core program or it may be written as a standalone plugin. That feature then becomes part of the software and everyone benefits. As a user of the software, you can easily contact someone involved in developing the software and ask questions and request features. With proprietary software you never have such direct access to the development team. You can also donate to the QGIS project. Your donation will pay for developers to fix bugs and implement new features. Beyond programming there are many ways people can contribute to a project like QGIS. You can report bugs when you encounter them, write and translate documentation, contribute a case study, and write books.SummaryOpen source is both a development methodology and a software license. In the end it is really impossible to say that a proprietary software like ArcGIS is better than the open source equivalent like QGIS or vice versa. You must decide if either works for you, and freedom and monetary cost may be part of that decision.End Note:There are a lot of acronyms in GIS and specifically open source GIS. Here are some you may encounter:FOSS = Free and Open Source. Historically there were two similar software movements, Free Software and Open Source software. They are so similar that they now are lumped together, and people simply use this acronym when talking about them.FOSS4G – Free and Open Source for Geospatial. This is free and open source software specifically for mapping. OsGeo (below) holds annual open source GIS conferences called FOSS4G. The next is coming up later this month in Bonn, Germany.OsGeo – Open Source Geospatial Foundation – this is a non-profit organization whose mission is to foster global adoption of open geospatial technology by being an inclusive software foundation devoted to an open philosophy and participatory community driven development. To be considered an OsGeo project a software must meet certain requirements. QGIS is a project under the OsGeo umbrella.

Field Data Collection with Fulcrum

Fulcrum was reviewed in our initial survey of field data collection apps in 2012, and almost made the top three. In the last 3 years Fulcrum has improved and has become perhaps the most intuitive and useful data collection app we've evaluated period.It is available for both iOS and Android. It isn't free, but the subscription fee is affordable. It costs anywhere from $18 - $25 per month. The three pricing plans give you 10 - 30Gb of online storage, which is substantial. Fulcrum offers a free 30 day trial which includes all the functionality. You can use this option to test Fulcrum for your projects. In the following example, I will be using a health care facility data collection form to show how Fulcrum works.Fulcrum has the most intuitive data collection form builder of any app we've seen. When you design a form Fulcrum calls it an 'app'. Simply drag and drop from the Add Fields section to your 'app' to add questions (see figure below). Available data input types include text, numbers, date, single or multiple choice, photos, videos, and audio.  There are no tricks to collecting GPS locations as with iForm. Fulcrum collects locations automatically.Once a field has been added simply set you parameters. The figure below shows the facility type question being edited. To do this simply click on a field, and fill out the details. It's so easy a 50 year old can do it!The companion mobile app can be downloaded for free from the Apple Store or the Google Play Store. Once installed, login and your data collection app(s) will sync with your mobile device. The figure below shows the health care facilities data collection app on an iPhone. Answering the questions is intuitive. Once collected your data will be synced with your cloud account.Once back in the office, login to your account, select your data collection app, and choose Start Export Wizard. You will be taken to the page below. Choose your file format. A complete array of GIS formats is available including: shapefiles, geodatabases, KML, PostGIS and Spatialite.  Choose any other appropriate options and click Next to download your data.I highly recommend that everyone involved in Community Health Mapping evaluate Fulcrum. Along with iForm and ODK Collect is a CHM recommended data collection tool. There is a monthly subscription fee but it is low. It is the easiest and most flexible tool we've found. You can use the free 30 day trial period to see if it works for you.

Field Data Collection with iForm

Unfortunately the most recent iOS system update rendered the EPI Collect app unusable. Apparently it is no longer being supported on the Apple platform. With this discovery, and a training in Charleston just around the corner, we set out to find a replacement. We searched for another free app for iPads and iPhones that allows you to develop your own data collection form. Fortunately we discovered iForm  which turns out to be even easier to use, and more robust. (NOTE: It is also available for Android devices.)This app has a lot of similarities with ODK Collect which we recommend for Android users (ODK Collect is described in the Field Data Collection blog post). With iForm you create a free account on the companion iFormBuilder website. You use their online form builder to create your data collection form. The form builder has over 30 different types of data inputs to choose from! For example: text, number, date, time, pick list, phone number, location (GPS coordinates) and images (photographs). Individual data elements can be set up as questions for the data collectors such as: What is the name of the site?Once the form is developed you can begin to collect data.

• Open the app on your mobile device and login.
• Tap the Sync button and all the forms and records that are associated with your account will be downloaded to the device.
• Head out to your project site and collect data.
• At the first data collection site simply open the data collection form, answer each question, and click Done to save the information.
• Repeat at each site.

If you are collecting data while in cellular coverage, the data will be synced to your iFormBuilder cloud account as you go. If you leave cellular coverage that is OK. The on-board GPS receiver on your mobile device will still allow you to collect your locations. Once you are back within cellular range you can Sync your data to your iFormBuilder cloud account. The data can be viewed on the mobile device in tabular or map format. Back in the office the data can be downloaded from the iFormBuilder site in several formats, the most useful of being an Excel spreadsheet. The data in the spreadsheet can then be brought in QGIS or CartoDB and mapped.iForm has some additional features that stream line data collection. You can link your iFormBuilder account to a DropBox or Box account. With this link established your data and photos will be uploaded to a DropBox folder automatically. There are also tools for assigning a form to different users. This allows you to develop one data collection form and share that among a team of data collectors.The free iFormBuilder account has some limits.  You are limited to 10 forms and 100 records per form. However, you can log in to your account, export the data, and delete those online records and continue data collection.In summary, iForm is a powerful and intuitive free app for collecting community health data with iPhones, iPad, and Android devices.

Technology + Youth = Change

by Chad Noble-TabioloIt all started in May 2013 when I watched the documentary entitled Revolutionary Optimists on PBS’s Independent Lens. It showed how young people from a slum in Kolkata, India were able to map the deficient and unsafe water taps in their community, in order to plea with the government for more and safe drinking water lines. The film highlighted technology in an unconventional way. It showed GIS-technology as an innovative tool to mobilize youth for social change.This heralded the beginning of a partnership with Map Your World to develop a mapping project in the Philippines in the summer of 2013. Through coordination with domestic and international partners, the youth mapping program was implemented in Southville 7 — an impoverished and neglected slum community, about three-hours south of metro Manila. The issues faced in Southville 7 ranged from lack of access to jobs, water and electricity to food insecurity and child and maternity health; and because of a lack of response from both the government and non-governmental sectors, the project was aimed to raise awareness and demand change.In just a few weeks, a dozen phones were donated. Youth, ranging from 15 to 23 years old, were trained to go house-to-house to collect data. By the end of three months, 3000 families were surveyed and the needs of the community were mapped. Depicted below is Map 1, which shows the families who have direct access to water in their homes.Because of the unequal distribution of resources, it was evident who had direct access to water and who did not. Map 2 shows those families who did not have direct access to water. These families had to walk more than 1 kilometer to a communal water tap.Lastly, Map 3, represents the top four needs according to the three different subdivisions or sites in Southville 7.  Collectively these maps and data provide an opportunity for proper and adequate planning for public health infrastructure and needs.The Android mobile phones used by the youth were powered by open-source applications for GPS-mapping and data collection. ODK Collect or Open Data Kit was the data collection tool utilized in the project. It can be found on the Android market. (NOTE: This tool is also described in the Community Health Mapping blog post on Field Data Collection). This tool is functional only after uploading a survey form that is created in Microsoft Excel and uploaded to the companion site www.formhub.org. The maps were created online with Map Your World, an online community mapping tool inspired by the Revolutionary Optimists documentary.In the end, the 30 youth involved in the mapping project were able to accomplish an endeavor that many people in their community had not expected. They were able to successfully map who in their community had access to water, electricity, jobs and vaccination for children under five years old, among others. They became leaders who are now equipped with leadership and technological skills that many in their community lack. They were empowered to raise awareness about the social injustices and health inequalities existing among them.The Southville 7’s mapping work was primarily a vehicle for instilling hope, and the use of GPS/mapping-technology offered an opportunity for the youth to be the voice for their community. According to one youth, “For me, mapping is like knowing. Knowing the problems, and how people are coping with them. Through the work we can open the eyes of the people, not only the things that can help them, but things that can help us all.”