Category: Open

mixing technologies – weekend edition

whbeebe

So. It’s late Saturday night, and here I sit in front of my little hackerish Ubuntu notebook. The clock just rang out midnight, meaning I’m rolling with this into Sunday, and I’m typing away while listening to Depech Mode’s “The Best Of” on Youtube that somebody in Russia ripped and uploaded. Yeah.

It’s been a long three or four days since I first wrote about starting out with Open Street Map, and it’s like I’ve been in a long, drawn-out fist fight, where the punches keep coming in from out of nowhere and I can’t land any of my own. That map above, with the southern states in place from Georgia west to Arizona, took me the majority of the time figuring out how to load.

Now, I haven’t been spending all this time on this task; I do have A Real Job. I’d take time off during a lunch and spend about 30 minutes on it, then spend another hour after getting home and after everything was done for the evening before hitting the sack. Even so, I never expected this kind of slow going.

Biggest problem is in the directions. Remember that page that was so helpful with its simple directions? Turns out it was too simple, and here’s why. At one point you need to load a file using the following command (this is for Florida, which is where I started to build my map of CONUS):

osm2pgsql –slim -C 1500 –number-processes 4 florida-latest.osm.pbf

That command loads the latest Florida map into the database. So far, so good. The problem is that if I use the same command to load any other file, it wipes out any other data. That’s when I found out about the “–append” command line switch. Sweet. Just add that to the rest of the commands to load the rest of the states and I’m done. No, not sweet. When run, I’d get the following error message:

failed: ERROR: duplicate key value violates unique constraint “planet_osm_rels_pkey”

I tried all sorts of ways to get around this, including the implied method of importing all my map files at the same time by placing all the file names on the command line, separated by spaces. That didn’t work either. Finally, while cruising through this forum on github I found out you needed to drop the “–slim” command line switch. WTF? When I did that with the Georgia file, all the other files loaded, and when it was all done I had the basic map you see above. I noticed something interesting about the slim switch. If you read the documentation, it seems to imply that using slim is actually faster than not using it. Not on my system. Leaving slim off actually sped things up a bit.

Note that I got so desperate to load anything in CONUS that I downloaded the 5GB map file of the entire continental North America and attempted to load it. I downloaded that late Friday, and started it up around 6pm. It ran through the night, and continued until I killed the whole thing around 3pm Saturday. That’s when I gave Google another stab at trying to find a solution. I did try DuckDuckGo, but it failed spectacularly in trying to find a solution; it gave exactly two (yes, I said two) hits. Google gave me page after page, and it was literally the first entry that gave up the vital clue about the use of slim. Google 1, DuckDuckGo 0.

Other issues I’ve had with my Open Street Map stack is getting it to generate all the necessary tiles at the default resolutions. I’ve been doing it the hard way with “/dirty” on the end of the map graphic. For example, in Chrome, selecting “Open Image in New Tab” gives you a new tab with “http://localhost/osm/5/7/12.png” in the address bar. Modify the URL to “http://localhost/osm/5/7/12.png/dirty” and hit return, and OSM responds with “Tile submitted for rendering“, and after a bit of time it will be rendered. I did this on a lot of blank tiles, then setting Permalink on that section and hitting refresh until it showed up. This is a royal pain, and no amount of searching calls out clearly what tool needs to be installed and/or run to generate a series of default tiles. I will find out how to do this, as my very labor intensive method sucks dead hamsters through a garden hose.

Here’s a few more screen shots showing the detailed maps I have so far.

The browsers used are, from top to bottom, Opera showing Austin Texas (West Sixth Street and the Colorado River), Firefox showing Orlando Florida (I-4/408 interchange and Lake Eola), and the JavaFX in Java browser showing the states of Tennessee, North and South Caroline, Georgia and Alabama. I’ve started to hack the original JavaFX demo application, having reversed the tab order and setting the default URL to point to the OSM slippymap page. The next step is to build a PostgreSQL viewer into the second tab so I can browse the GIS tables that hold the OSM map data. Before I do anything else I’m dumping the database to back it up. I want to add the rest of the continental US states, and maybe a few of the Canadian cities like Toronto.

While I was at it I also pointed my Nexus 7 2012 tablet, running KitKat, at the OSM stack, just to see how it rendered on an Android tablet. I have my reasons. My systems are wirelessly connected so it wasn’t any big deal to punch in the IP of the Ubuntu system along with the default slippymap URL and give it a go.

Both of these screenshots were using Opera. I have to say that I really enjoy using Opera on my Samsung Galaxy S4 (Android 4.3), Nexus 7 2012 (Android 4.4.2) and even my Barnes & Noble Nook HD+ (Android 4.0). It seems to be fast and use a minimal amount of memory. I like the Speed Dial page, especially on the phone. The only problem I had rendering my local OSM map was with Firefox, as you can see below.

I don’t know what’s happening here, but dragging the map around caused it to tear apart. I don’t know why that happens, but I won’t lose any sleep over it. Firefox does just fine with Google Maps and the commercial implementation of OSM, so it’s definitely something in the current installation. Since I need to make all Javascript references on slippymap local, I hope dropping the latest versions of OpenLayers and OpenStreetmap will fix that issue. Speaking of commercial implementation, I came across this courtesy of DuckDuckGo:

Yes, it’s MapQuest Open, but this site is using Open Street Map data as explained here. Looks kinda pretty, don’t it? Anyway, that’s it for the evening. It’s now past 1 AM, and I’ve had enough excitement for one weekend.

open street map in a box

whbeebe

Out of all the many personal projects I’ve started these days, one over-arching project is learning how to live without Google and its many myriad services. Google’s search, maps and email are the most widely known. Google’s trying to get into The Social via Google+, and it’s established a nearly unassailable position in mobile (smart phone and tablet) operating systems with Android. Google’s stock market valuation is in the hundreds of billions and its quarterly earnings are in the tens of billions. Google has the resources to hire the best and brightest and to keep them. It refines what it has and is trying to develop new areas for future expansion, such as AI, robotics, and home automation (see Nest acquisition). Its offerings are broad, deep, and refined.

The problem is the cost to use these goods and services, and that’s Google’s increasingly invasive collection of personal information. In the past I tended, like a frog in a slowly heating pot, to not pay too much attention to the loss of my personal privacy and information as the cost of using these goods and services “for free.” All that changed with the NSA revelations, and the realization that no matter how much Google, Facebook,  Microsoft et el righteously complain, they helped the NSA by providing methods, tools, and eventually that same personal information. I know that NSA is trolling for far more information, such as cellular phone metadata, but there would be no metadata (or data in general) if it wasn’t collected by commercial interests in such detail in the first place.

With that background, I’ve been looking at open source tools and alternative services to slowly replace what Google currently offers. In order to make this work to the largest degree possible, I’ve come to the realization that I may not match every feature that Google has to offer, and that’s all right by me. I’ll try to replace a feature completely if I can with an open alternative. If it doesn’t exist or it’s in a form that is incomplete, I’ll attempt to bring it to parity with a Google feature. And if I can’t do that, then I’ll simply learn to live without. Personal privacy and freedom are far more important than convenience with chains attached.

I started looking for a substitute to web-based Google Maps. There’s been a substitute out there for a long time, Open Street Map. Tonight I navigated over to SWITCH2OSM and clicked on the link that led to a single page, showing step-by-step how to install the OSM tools and bring them up on an Ubuntu system. The OSM tools can be installed as a series of packages on Ubuntu version 12.04 (LTS) and higher. My Samsung R580 is running 13.10. By following the very clear directions on that page I was able to install and start my own OSM map server. The only problem I ran into was self-inflicted: I’d installed nginx for an earlier project and forgot it was running. The last stage in installing the OSM tools is to restart the Apache web server as a service. That restart failed because the HTTP port was already being used by nginx. I uninstalled nginx and restarted the Apache service by hand, and everything started to work. Since then I’ve been poking at it and pulling down some of the smaller tiles, starting with the state of Florida. It took less than an hour to get a bare but reasonably functional stand-alone OSM system up. I’ll now be using it to serve tiles to my other notebook for development purposes.

The first two screen shots show Orlando, first with OSM and then with Google Maps. What should be immediately apparent is the lack of controls on the OSM screen. Specifically, there’s no search bar to look for a map location. Since this is my absolute first foray into OSM, I’m not at all concerned. The URL provided is a check-out capability to just see if the whole OSM software stack is working, and it is, with flying colors.

Another difference is visual on the maps themselves. The OSM map has a lot more detail at the same resolution. Levels of detail and how they’re presented are very personal, and once again can be programmatically tweaked in the page itself as well as with some work on the backend. I’m sure there are other open source web pages that can provide the features, and if they aren’t, well, then, dig in and create them and provide them back to the OSM community.

These last two screen shots are of Toronto. I did not download any specific extracts for Toronto or Canada. What you see is what was installed with the initial OSM software stack, and it appears to be quite detailed as-is. Once again the OSM page is missing all the Google Map widgets, and truth be told, I like the clean minimalism of the page. Google’s widgets now encroach too much into the information provided, especially the bar across the top.

I don’t know where I’m going with this, as creating your own map (tile) server can be seen as a bit of overkill. After all, you can go to Open Street Map and just use it. But I’m an inveterate tinkerer; for me it’s interesting to look beneath the “surface” and see how it all works, and even make your own changes, just for the hell of it. It’s all open and inviting.