Ghoscher's Blog

Hello, welcome to the 6th part of my tutorial on how to create an offline? map app without using Google™ API, okay this time it’s no longer offline :D! now the app will be able to download tiles from the internet :D!.

Series outline:

• Series aim.
• Theory you need to know.
• App design.
• Writing a TilesManager.
• Writing a TilesProvider.
• Seeing results with MapView.
• Adding web support
• Creating MapView in XML.
• Extra: Fuchs Maps.

______________________________________

At first I planned on writing some basic tips about web tiles server support but then I realized that implementing web support needs some work and code organizing since multi threading becomes a must!. You cannot wait for the tiles to be downloaded in the UI thread!.

We will assume that you have a running tiles server that provides you with tiles you specify using a custom url like this:

http://myserver.com/?x=1&y=3&z=0
or

When we want a specific tile we first search if we have it in our memory cache, if we don’t then we search in the database, if we fail to find it then we download it from the internet.
It’s kind of similar to multi cache levels in a cpu, if one level fails it gets the requested data from the next level, when data found in a certain level all lower levels save a copy of the data since it’s highly likely to be asked for the same data in the near future.
In our case when you don’t have a tile in memory or in database you get it from the web server and then save it in the database and keep it in memory for later use.

We’ve already implemented the first two levels (Memory & database) in the tutorial part 4, the TilesProvider class covered them both. but we didn’t have any threading manipulation, this time we have to be very careful about threading stuff. We will implement new classes and modify some of the old ones, the changed lines in existing classes will be highlighted in grey. Also note that if you’re using Eclipse as your IDE you can always count on the keyboard shortcut Ctrl+Shift+O to automatically import the right packages.


We will add three source files to our project in a new package called com.mapapp.tileManagement.web

Let’s first implement the class TileDownloadTask, this class handles a single download request, we simply pass it the url of the file we want to download, it doesn’t handle any multi threading, it just implements the Runnable interface so that it can be executed from the outside in a separate thread, more on that later :).

Here’s the constructor of the class TileDownloadTask:

public TileDownloadTask(String myUrl, DownloadTaskFinishedCallback callback, int x, int y, int z)

The first parameter is obviously the tile to download, the second is a simple interface we’ll define later, we use it inform whoever interested that this task has finished.

The last three parameters are just to tell which TileDownloadTask is which, because we will execute more than one download task and all of them will report their results to the same callback. here’s the complete code for the interface DownloadTaskFinishedCallback:
DownloadTaskFinishedCallback.java

package com.mapapp.tileManagement.web;

public interface DownloadTaskFinishedCallback
{
	public void  handleDownload(TileDownloadTask task);
}

Okay back to the TileDownloadTask, the class will simply download the requested tile then pass itself (using this) to the callback provided in the constructor, here’s the full class code, please read the comments, it’s easier to explain with comments:

TileDownloadTask.java

package com.mapapp.tileManagement.web;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.SocketTimeoutException;
import java.net.URL;

public class TileDownloadTask implements Runnable
{
	// Task state constants
	public final static int TASK_ONGOING = 0; // Starting or ongoing
	public final static int TASK_COMPLETE = 1; // Completed successfully
	public final static int TASK_FAILED = 2; // Failed for some reason

	private final String myUrl; // Url of the tile to download

	// Coordinates of the tile being downloaded, needed later
	private final int x, y, z;

	// Called when task finishes its work or when it fails.
	private final DownloadTaskFinishedCallback callback;

	// Current task state
	private int taskState = TASK_ONGOING;

	// Will contain the downloaded file
	private byte[] file = null;

	public TileDownloadTask(String myUrl, DownloadTaskFinishedCallback callback, int x, int y, int z)
	{
		this.myUrl = myUrl;
		this.callback = callback;

		this.x = x;
		this.y = y;
		this.z = z;
	}

	// Download code goes here
	public void run()
	{
		try
		{
			// See here for explanation
			// http://developer.android.com/training/basics/network-ops/connecting.html#download
			URL url = new URL(myUrl);
			HttpURLConnection conn = (HttpURLConnection) url.openConnection();

			// Time in milliseconds the task has to download the tile
			conn.setReadTimeout(10000);
			conn.setConnectTimeout(15000);
			conn.setRequestMethod("GET");
			conn.setDoInput(true);
			conn.connect();

			int response = conn.getResponseCode();
			if (response == 404) // Not found
			{
				taskState = TASK_FAILED;
				return; // Go to finally block
			}

			// Reading file
			InputStream is = conn.getInputStream();
			byte[] buffer = new byte[1024 * 4];
			ByteArrayOutputStream out = new ByteArrayOutputStream();

			while (true)
			{
				int read = is.read(buffer);
				if (read == -1) break;

				out.write(buffer, 0, read);
			}

			out.flush();
			file = out.toByteArray();
			out.close();

			taskState = TASK_COMPLETE;
		}
		catch (SocketTimeoutException ste)
		{
			taskState = TASK_FAILED;
		}
		catch (MalformedURLException e)
		{
			e.printStackTrace();
			taskState = TASK_FAILED;
		}
		catch (IOException e)
		{
			e.printStackTrace();
			taskState = TASK_FAILED;
		}
		catch (Exception e)
		{
			taskState = TASK_FAILED;
		}
		finally
		{
			// Report the result by passing this task
			if (callback != null) callback.handleDownload(this);
		}
	}

	// Simple getters

	public String getUrl()
	{
		return myUrl;
	}

	public byte[] getFile()
	{
		return file;
	}

	public int getX()
	{
		return x;
	}

	public int getY()
	{
		return y;
	}

	public int getZ()
	{
		return z;
	}

	public int getState()
	{
		return taskState;
	}
}

The class implements the Runnable interface, this way we can create a new TileDownloadTask and run it in its own thread easily!.

You’ll notice that we specified some timeouts in the download code, we don’t want a download request to take forever (due to some error connecting maybe!?), so the task either downloads the file in time or is considered a failure. At the end of the run() method we make sure to report the result using the callback in the finally block.

The class also contains some getters to be used from the inside to retrieve the tile data and its coordinates.

We now have a class that handles a single download but doesn’t involve threading, the next step is to write class that will handle multiple download requests and put each one in a separate thread.

The way we’re going to do this is by creating a new class called WebTilesProvider, it implements the DownloadTaskFinishedCallback interface, just for the TileDownloadTask to be able to inform it when it has finished the download

public class WebTilesProvider implements DownloadTaskFinishedCallback

it will have the following methods:

public WebTilesProvider(int threadsCount, DownloadTaskFinishedCallback handler)

Constructor first parameter defines the maximum number of threads used for downloading the requested tiles, second parameter is a simple interface that we use to pass the downloaded tiles to.

public void downloadTile(int x, int y, int z)

Creates and executes a single download task.

How this class works is as follows: it has a set of unfinished requests URLs (I called pendingRequests), whenever we request a download it checks if it already has this request in that set, if the request is new (not found in set) then it creates a new TileDownloadTask with the specified url and executes it in a new thread.

For threading I used an ExecutorService, this class does a great job! you can just give it runnables and it takes care of executing each one in a separate thread, here we create the ExecutorService with fixed number of threads, which means the maximum number of tasks being executed is fixed, other tasks will have to wait for their turn.

In multithreaded applications there are some code blocks that you don’t want them to be accessed by more than a thread at a time, in our case we don’t want a thread to add a request to the set while another thread is deleting something from the set, such things will cause exceptions and instability in the application, to solve this we use the keyword synchronized on those sensitive code blocks, the keyword needs and object to function, it can be any object, just make sure that you don’t change the object by assigning it some other object. Note that those code blocks could be in more than one method\class, as long as you’re using the same object for synchronizing it will work!.

The keyword synchronized also works with methods, this prevents two threads from accessing the same method together.

Back to WebTilesProvider here’s the full class:

WebTilesProvider.java

package com.mapapp.tileManagement.web;

import java.util.HashSet;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class WebTilesProvider implements DownloadTaskFinishedCallback
{
	// Max number of active download threads
	// A large number of threads might get
	// you blocked from the tiles server.
	final int threadsCount;

	// Keeping track of current non-finished tasks
	// to avoid downloading a tile more than once
	HashSet pendingRequests = new HashSet();

	ExecutorService pool; // Handles requests

	// A callback to be called by finished\failed tasks
	DownloadTaskFinishedCallback handler;

	public WebTilesProvider(int threadsCount, DownloadTaskFinishedCallback handler)
	{
		this.threadsCount = threadsCount;
		pool = Executors.newFixedThreadPool(threadsCount);

		this.handler = handler;
	}

	public void downloadTile(int x, int y, int z)
	{
		// Get the url in the right format
		String url = formatUrl(x, y, z);

		// Whenever using the HashSet pendingRequests we must
		// make sure that no other thread is using it, we do that by
		// using synchronized on the set whenever a code block uses the set
		synchronized (pendingRequests)
		{
			// If tile isn't being downloaded then add it
			if (!pendingRequests.contains(url))
			{
				pendingRequests.add(url);

				// Create a new task and execute it in a separate thread
				TileDownloadTask task = new TileDownloadTask(url, this, x, y, z);
				pool.execute(task);
			}
		}
	}

	String formatUrl(int x, int y, int z)
	{
		// Here we're using open street map tiles, you can replace it with the
		// server you want
		// Just make sure you have the right to download the tiles
		// Also note the zxy order for the tiles!
		String result = String.format("http://a.tile.openstreetmap.org/%s/%s/%s.png", z, x, y);

		return result;
	}

	// This function should be called when the TilesProvider has
	// received and processed the tile, it should be called even when the
	// download fails, otherwise
	// the request will be stuck in pendingRequest without actually being
	// executed!
	// leaving the tile blank.
	private void removeRequestFromPending(String url)
	{
		// Making sure no other thread is using the set
		synchronized (pendingRequests)
		{
			pendingRequests.remove(url);
		}
	}

	// Called by a TileDownloadTask when finished
	@Override
	public synchronized void handleDownload(TileDownloadTask task)
	{
		int state = task.getState();

		// If downloaded successfully
		if (state == TileDownloadTask.TASK_COMPLETE)
		{
			// Pass the task to the TilesProvider
			if (handler != null) handler.handleDownload(task);
		}
		else if (state == TileDownloadTask.TASK_FAILED)
		{
			// Do nothing!!
		}

		// It's necessary to remove the request from pending list
		// We only remove it when we are done with it, otherwise the MapView
		// could request the tile while it's being inserted in the database for
		// example.
		// This way we make sure we download the tile only once.
		removeRequestFromPending(task.getUrl());
	}

	// Hopefully kills the active download tasks and clears all pending tasks
	public void cancelDownloads()
	{
		pool.shutdownNow();
		synchronized (pendingRequests)
		{
			pendingRequests.clear();
		}

		// Cannot reuse ExecutorService after calling shutdownNow
		// Create a new executor
		pool = Executors.newFixedThreadPool(threadsCount);
	}
}

If the comments weren’t enough here’s a little more explanation:

pendingRequests is a set containing all the URLs of the download requests that haven’t yet finished.

handler is another DownloadTaskFinishedCallback and it is passed in the constructor, the work flow for downloading and processing tiles is like the following:

– TilesProvider asks WebTilesProvider to download a tile with x,y,z

-WebTilesProvider creates a TileDownloadTask with a url and x,y,z

-TileDownloadTask finishes the download and passes itself to the WebTilesProvider

-WebTilesProvider removes the request from pendingRequests using the url from the task and then passes the TileDownloadTask to TilesProvider

-TilesProvider gets the tile from the task and inserts it to the database with the index getX(),getY(),getZ() from the task.

-TilesProvider sends a message to the activity about the arrival of a new tile, the activity issues a redraw order to the MapView.

handleDownload(TileDownloadTask task) this method is called from inside the TileDownloadTask when its job is done, note that we remove the request from pendingRequests only when the TilesProvider is done with it.

I know this whole callbacks stuff might look confusing but it’s necessary, I could have merged the class WebTilesProvider with TilesProvider but I preferred to keep them separate by adding a callback in between.

Okay we’re done now with the new classes, we just have to modify the old classes a bit, we need to modify TilesProvider, MapView and MapAppActivity, so let’s start with TilesProvider.

Modifying TilesProvider:

The class will now implement the interface DownloadTaskFinishedCallback so that WebTilesProvider can pass the finished tasks to it.

public class TilesProvider implements DownloadTaskFinishedCallback

The fields will be like this

WebTilesProvider webProvider;

// The database that holds the map
protected SQLiteDatabase tilesDB;

// Tiles will be stored here, the index\key will be in this format x:y
protected Hashtable<String, Tile> tiles = new Hashtable<String, Tile>();

// An object to use with synchronized to lock tiles hashtable
public Object tilesLock = new Object();

// A handler from the outside to be informed of new downloaded tiles
// Used to redraw the map view whenever a new tile arrives
Handler newTileHandler;

We added a WebTilesProvider at the top, an Object called tilesLock that we will use with the keyword synchronized for code blocks that deal with the Hashtable tiles, and finally we added a Handler that we will use to inform the MapAppActivity about new downloaded tiles.

Note that here I’m using the android.os.Handler instead of some custom callback, that handler gives us the ability to communicate between threads safely, you don’t want to modify a view (MapView in our case) from a thread other than the UI tread, this handler is passed in the construcor by MapAppActivity.

Here’s the constructor

public TilesProvider(String dbPath, Handler newTileHandler)
{
	/*
	 *  Create WebTileProvider with max number of thread equal to five
	 *  We also pass this class as a DownloadTaskFinishedCallback
	 *  This way when the web provider downloads a tile we get it
	 *  and insert it into the database and the hashtable
	 */
	webProvider = new WebTilesProvider(5, this);

	// This time we are opening the database as read\write
	tilesDB = SQLiteDatabase.openDatabase(dbPath, null, SQLiteDatabase.OPEN_READWRITE);

	// This handler is to be notified when a new tile is downloaded
	// and available for rendering
	this.newTileHandler = newTileHandler;
}

Now we must modify fetchTiles to download missing tiles, what will happen is that we use the rectangle passed to the function to make a list of all the requested tiles (expectedTiles), and then we search for tiles in the Hashtable tiles and in the database.

Next we remove any tile we have from the expected tiles list, the next step is to download all the tiles we are missing by iterating through the list expectedTiles and downloading each tile.

Here’s the code for the method fetchTiles

// Updates the tiles in the hashtable
public void fetchTiles(Rect rect, int zoom)
{
	// We are using a separate object here for synchronizing
	// Using the hashtable tiles will cause problems when we swap the
	// pointers temp and tiles
	synchronized (tilesLock)
	{
		// Max tile index for x and y
		int maxIndex = (int) Math.pow(2, zoom) - 1;

		// First we create a list containing the index of each tile inside
		// the rectangle rect, we're expecting to find these tiles in memory
		// or
		// in the database
		ArrayList expectedTiles = new ArrayList();
		for (int x = rect.left; x <= rect.right; x++)
		{
			// Ignore tiles with invalid index
			if (x < 0 || x > maxIndex) continue;
			for (int y = rect.top; y <= rect.bottom; y++)
			{
				if (y < 0 || y > maxIndex) continue;
				expectedTiles.add(x + ":" + y);
			}
		}

		// Perpare the query for the database
		String query = "SELECT x,y,image FROM tiles WHERE x >= " + rect.left + " AND x <= " + rect.right + " AND y >= " + rect.top
					+ " AND y <=" + rect.bottom + " AND z == " + (17 - zoom); 		// query should be something like: 		// SELECT x,y,image FROM tiles WHERE x>=0 AND x<=4 AND y>=2 AND
		// y<=6
		// AND
		// z==6

		Cursor cursor;
		cursor = tilesDB.rawQuery(query, null);

		// Now cursor contains a table with these columns
		// x(int)	y(int)	image(byte[])

		// Prepare an empty hash table to fill with the tiles we fetched
		Hashtable<String, Tile> temp = new Hashtable<String, Tile>();

		// Loop through all the rows(tiles) of the table returned by the
		// query
		// MUST call moveToFirst
		if (cursor.moveToFirst())
		{
			do
			{
				// Getting the index of this tile
				int x = cursor.getInt(0);
				int y = cursor.getInt(1);

				// Try to get this tile from the hashtable we have
				Tile tile = tiles.get(x + ":" + y);

				// If This is a new tile, we didn't fetch it in the
				// previous
				// fetchTiles call.
				if (tile == null)
				{
					// Get the binary image data from the third cursor
					// column
					byte[] img = cursor.getBlob(2);

					// Create a bitmap (expensive operation)
					Bitmap tileBitmap = BitmapFactory.decodeByteArray(img, 0, img.length);

					// Create the new tile
					tile = new Tile(x, y, tileBitmap);
				}

				// The object "tile" should now be ready for rendering

				// Add the tile to the temp hashtable
				temp.put(x + ":" + y, tile);
			}
			while (cursor.moveToNext()); // Move to next tile in the
												// query

			// The hashtable "tiles" is now outdated,
			// so clear it and set it to the new hashtable temp.

			/*
			 * Swapping here sometimes creates an exception if we use
			 * tiles for synchronizing
			 */
			tiles.clear();
			tiles = temp;
		}

		// Remove the tiles we have from the ones to download
		for (Tile t : tiles.values())
		{
			expectedTiles.remove(t.x + ":" + t.y);
		}

		// Download the tiles we couldn't find
		for (String string : expectedTiles)
		{
			int x = 0, y = 0;
			String[] nums = string.split(":");
			x = Integer.parseInt(nums[0]);
			y = Integer.parseInt(nums[1]);
			webProvider.downloadTile(x, y, zoom);
		}
	}
}

You should notice that the whole method is inside synchronized(tilesLock), that’s because I don’t want any other thread messing with the tiles, that’s why I use synchronized(tilesLock) whenever I want to use the Hashtable tiles!.

Note that tilesLock is public as it’s needed out side the class.

Next is the method clear(), we now need to cancel all downloads and pending tasks too.

public void clear()
{
	// Make sure no other thread is using the hashtable before clearing it
	synchronized (tilesLock)
	{
		tiles.clear();
	}

	// Cancel all download operations
	webProvider.cancelDownloads();
}

Finally we have two new methods

//Called by the WebTilesProvider when a tile was downloaded successfully
//Also note that it's marked as synchronized to make sure that we only handle one
//finished task at a time, since the WebTilesProvider will call this method whenever
//a task is finished
@Override
public synchronized void handleDownload(TileDownloadTask task)
{
	byte[] tile = task.getFile();
	int x = task.getX();
	int y = task.getY();

	// Log.d("TAG", "Downloaded " + x + ":" + y);

	// Insert tile into database as an array of bytes
	insertTileToDB(x, y, 17 - task.getZ(), tile);

	// Creating bitmaps may throw OutOfMemoryError
	try
	{
		Bitmap bm = BitmapFactory.decodeByteArray(tile, 0, tile.length);
		Tile t = new Tile(x, y, bm);

		// Add the new tile to our tiles memory cache
		synchronized (tilesLock)
		{
			tiles.put(x + ":" + y, t);
		}

		// Here we inform who ever interested that we have a new tile
		// ready to be rendered!
		// The handler is in the  MapAppActivity and sending it a message
		// will cause it to redraw the MapView
		if (newTileHandler != null) newTileHandler.sendEmptyMessage(0);
	}
	catch (OutOfMemoryError e)
	{
		 // At least we got the tile as byte array and saved it in the
		 // database
	}
}

// Marked as synchronized to prevent to insert operations at the same time
synchronized void insertTileToDB(int x, int y, int z, byte[] tile)
{
	ContentValues vals = new ContentValues();
	vals.put("x", x);
	vals.put("y", y);
	vals.put("z", z);
	vals.put("image", tile);
	tilesDB.insert("tiles", null, vals);
}

We’re almost there, we just need to modify MapView and MapAppActivity a little.

Modifying MapView:

In MapView we simply use a synchronized block in the method drawTiles using the tilesLock object from the TilesProvider:

void drawTiles(Canvas canvas, Point offset)
{
	/*
	 * We use the same object in the TilesProvider when drawing
	 * This is necessary to make sure no one changes the available tiles
	 * while the MapView is being rendered
	 */
	synchronized (tileProvider.tilesLock)
	{
		// Get tiles from the Hashtable inside tilesProvider
		Collection tilesList = tileProvider.getTiles().values();

		// x,y are the calculated offset

		// Go trough all the available tiles
		for (Tile tile : tilesList)
		{
			// We act as if we're drawing a map of the whole world at a
			// specific
			// zoom level
			// The top left corner of the map occupies the pixel (0,0) of
			// the
			// view
			int tileSize = tileManager.getTileSize();
			long tileX = tile.x * tileSize;
			long tileY = tile.y * tileSize;

			// Subtract offset from the previous calculations
			long finalX = tileX - offset.x;
			long finalY = tileY - offset.y;

			// Draw the bitmap of the tiles using a simple paint
			canvas.drawBitmap(tile.img, finalX, finalY, bitmapPaint);
		}
	}
}

Modifying MapAppActivity:

we define a new android.os.Handler that will redraw the MapView when it receives a message from TilesProvider about new available tiles.

/*
 * A message is sent from TilesProvider when
 * a new tile is downloaded and ready for rendering
 * The tile should be stored in the memory cache we have
 * so all we have to do is to redraw the MapView
 */
Handler newTileHandler = new Handler()
{
	// This is executed on the UI thread
	public void handleMessage(android.os.Message msg)
	{
		// Ask the mapView to redraw itself
		if (mapView != null) mapView.invalidate();
	};
};

Then we update the line where we create the TilesProvider, we supply newTileHandler as a second parameter

void initViews()
{
	// Creating the bitmap of the marker from the resources
	Bitmap marker = BitmapFactory.decodeResource(getResources(), R.drawable.marker);

	// Creating our database tilesProvider to pass it to our MapView
	String path = Environment.getExternalStorageDirectory() + "/mapapp/world.sqlitedb";
	tilesProvider = new TilesProvider(path, newTileHandler);

	// Creating the mapView and make sure it fills the screen
	Display display = getWindowManager().getDefaultDisplay();
	mapView = new MapView(this, display.getWidth(), display.getHeight(), tilesProvider, marker);

	// If a location was saved while pausing the app then use it.
	if (savedGpsLocation != null) mapView.setGpsLocation(savedGpsLocation);

	// Update and draw the map view
	mapView.refresh();
}

New Permissions:

We just need to add the right permissions for the new features, the internet access and storage write (for the database) in the AndroidManifest.xml

Aaaaand we’re done :D!.

Hopefully you followed this step by step tutorial and you got a running exception-free app :D, if not here’s the full final soruce code along with the apk, in this tutorial the database provided is an empty one, just to make sure everything is running as it should:

MapApp_Final_Web Source Code.zip (51KB)

Map App Final Web.apk (20KB)

World.sqlitedb (15KB) (Please put in sdcard/mapapp/)

 

When you run the app for the first time and zoom in using the volume keys you will notice that the tiles will take some time to appear, the next time they will appear much faster! since the tiles were inserted in the database.

Final Notes:

• In this tutorial I used the Open Street Map tiles server, here’s their copyright and license page.
• Here’s a tutorial about concurrency/multi threading in Java
• You can find the code of RMaps here, it has more advanced features.
• The code here isn’t perfect, it’s not meant to be perfect, the main purpose of this tutorial is to provide a basic idea, you might need to do more exception checks maybe more code organizing yourself.
• I’ll be happy to answer questions :).