Using gzip compression when making HTTP requests in Windows Phone apps

Converge

Bandwidth and data consumption on mobile devices remain one of the most critical resources an app consumes on any platform. A majority of users have limited data plans and actually pay more if they use more data on their mobile devices. This makes it critical for us, as app developers, to ensure our apps consume as less data as possible. And when apps download less data, they perform better and become more responsive too.

I overlooked this myself earlier but fixed it as soon as I noticed. In fact, a lot of apps is store are not using compression including some heavily used apps like famous YouTube clients. The response size for 10 entries in most popular list from YouTube is ~50KB by default and ~7.4KB when compression is used, that’s 85% less data downlaoded. It varies for each request but the savings are somewhere from 30% to 85%.

Overview

We all use compressed archives on PCs in one format or the other like zip, rar, 7z and others. For the web, what it really means is that the data being transferred over the network is compressed and hence uses less bandwidth. This is defined in http spec as a standard. When a client, like your app, sends a request, it can tell the server that it’s capable of handling compressed response in Accept-Encoding request header. The server then sends the response in compressed format (if it’s capable of doing so) and tells the client using Content-Encoding response header. Obviously, the server and client both not only need to support this but also send consent that this capability is used. This is how http works for everything so no surprises there.

Good news is that you don’t have to implement compression to take advantage of this as the HttpClient library you should be using added supports for this over a year ago. Unfortunately, this wasn’t the case in early days of Windows Phone but workarounds existed. Ideally, the API should use compression by default but it doesn’t.

Clarification

The HttpClient I mentioned above is for Windows Phone 7.x and 8.x apps using Silverlight not WinRT. The .NET stack has had multiple APIs for making web request overtime but HttpClient is the one everyone should be using now which is wrapper/replacement for all other HTTP APIs in .NET framework.

Starting with WinRT which is used for developing “universal” apps, there is a separate HttpClient API. This seems to be using compression by default but please verify that it is. Here is a good starting point for HttpClient in WinRT and samples. If this all seems very confusing, you are not alone. The HTTP API in Microsoft world is a mystery wonderland of confusion so ask if you have questions.

Implementation

Using compression in your app is extremely simple and quick as explained by the framework team. When you create an instance of HttpClient class, send an instance of HttpClientHandler in constructor with appropriate setup and you are done. All http requests sent using this instance will now send correct request header and automatically un-compress the response.

var handler = new HttpClientHandler();
if (handler.SupportsAutomaticDecompression)
{
    handler.AutomaticDecompression = DecompressionMethods.GZip |
                                     DecompressionMethods.Deflate;
}

var client = new HttpClient(handler);

Performance Tip

While you are at it, here is another performance tip. When you are downloading data from internet and deserializing it, instead of first storing response as a string and then parsing it, parse data directly from the response stream. This works seamlessly for JSON as well as XML content.

For JSON, JSON.NET supports deserializing data from stream. It’s not as obvious as deserializing from a string though. Using the HttpClient instance we created above, here is a code snippet to deserialize response stream.

try
{
    using (var stream = await httpClient.GetStreamAsync(url))
    {
        using (var streamReader = new StreamReader(stream))
        {
            using (var textReader = new JsonTextReader(streamReader))
            {
                var serializer = new JsonSerializer();
                var data = serializer.Deserialize<List<Entry>>(textReader);
            }
        }
    }
}
catch (ObjectDisposedException ex)
{
    // ignore this exception as stream gets closed somehow and "using" block tries to dispose it again
    Debug.WriteLine(ex);
}

For XML response, XDocument supports parsing streams directly.

using (var stream = await httpClient.GetStreamAsync(url))
{
    XDocument.Load(stream);
}

For using stream with JSON.NET, I came across an issue though. As I am using “using” constructs to ensure the resources get cleaned up, the stream gets closed twice which causes an exception. To workaround that, ignore ObjectDisposedException exception as I have done above.

One drawback of using compression is that the phone will have to spend some CPU cycles to uncompress data. With today’s phones though, this is negligible. There is no reason to not use compression in your apps.

Hope this helps.

image credit: wpcentral

Notifying users when app update is available in Windows Phone store

Ensuring that users of your app are on latest version is critically important. I have seen users reporting issues and requesting features which were implemented a few updates ago. With Windows Phone 8.1, users can enable automatic app updates installation but for users who are still on Windows 8 or who do not enable automatic updates, the responsibility is on you, as a developer.

Unfortunately, there is no support in Windows Phone SDK to find out if there is a new version of the current app in store or what version is available in store. For the converge app, I was looking to solve this issue so I traced store app’s network traffic using Fiddler to understand the available APIs. Looking at the network traffic, the API used by store app is fairly obvious. Here is the API request Windows Phone 8 store app sends for app details.

GET http://marketplaceedgeservice.windowsphone.com/v8/catalog/apps/b658425e-ba4c-4478-9af3-791fd0f1abfe?os=8.0.10521.0&cc=US&lang=en-US&hw=520208901&dm=RM-940_nam_att_200&oemId=NOKIA&moId=att-us&cf=99-1
Accept: */*
Accept-Encoding: gzip
User-Agent: ZDM/4.0; Windows Mobile 8.0
X-WP-Client-Config-Version: 107
X-WP-MO-Config-Version: 1224
X-WP-Device-ID: ****
X-WP-ImpressionId: ****:API.BDIGeneric
X-WP-ImpressionK: 5

You don’t need to send all headers or parameters in requests. The response to this call has complete details for your application in the target country/language in xml format. From here on, you can use LINQ to XML to get the data you need to know what version is available in store. Here is slightly modified version of the code I use in the converge app. Feel free to reuse.

//This code has dependency on Microsoft.Net.Http NuGet package.
public async Task CheckUpdate()
{
    const string storeAppDetailsUri = "http://marketplaceedgeservice.windowsphone.com/v8/catalog/apps/b658425e-ba4c-4478-9af3-791fd0f1abfe?os={0}&cc={1}&lang={2}";

    var updatedAvailable = false;

    try
    {
        var osVersion = Environment.OSVersion.Version.ToString(4);
        var lang = CultureInfo.CurrentCulture.Name;
        var countryCode = lang.Length == 5 ? lang.Substring(3) : "US";

        using (var message = new HttpRequestMessage(HttpMethod.Get, string.Format(storeAppDetailsUri, osVersion, countryCode, lang)))
        {
            message.Headers.Add("User-Agent", "Windows Mobile 8.0");

            using (var client = new HttpClient())
            {
                var response = await client.SendAsync(message);
                if (response.StatusCode != HttpStatusCode.OK) return;

                using (var stream = await response.Content.ReadAsStreamAsync())
                {
                    XNamespace atom = "http://www.w3.org/2005/Atom";
                    XNamespace apps = "http://schemas.zune.net/catalog/apps/2008/02";

                    var doc = XDocument.Load(stream);
                    if (doc.Document == null) return;

                    var entry = doc.Document.Descendants(atom + "feed")
                        .Descendants(atom + "entry")
                        .FirstOrDefault();

                    if (entry == null) return;

                    var versionElement = entry.Elements(apps + "version").FirstOrDefault();
                    if (versionElement == null) return;

                    Version storeVersion;

                    if (Version.TryParse(versionElement.Value, out storeVersion))
                    {
                        var currentVersion = new AssemblyName(Assembly.GetExecutingAssembly().FullName).Version;

                        updatedAvailable = storeVersion > currentVersion;
                    }
                }
            }
        }
    }
    catch (Exception ex)
    {
        // HANDLE ERROR HERE. THERE IS NO POINT IN SHOWING USER A MESSAGE. GOOD PLACE TO SEND SILENT ERROR REPORT OR JUST SWALLOW THE EXCEPTION.
        Debug.WriteLine(ex);
    }

    if (updatedAvailable)
    {
        // APP UPDATE IS AVAILABLE. SHOW FIREWORKS IN APP OR JUST OPEN STORE APP TO UPDATE THE APP AFTER USER'S CONSENT.
    }
}

This is definitely not a documented and supported API  but considering there are millions of devices using this API, I doubt this will break in foreseeable future. I just don’t understand why Windows Phone team doesn’t make this API public and have folks build stuff on top of it. They have struggled to produce a good store experience since Windows Phone launched anyways. For Windows 8, the APIs are as useless as the store app is so don’t bother trying to do something similar there.

A side note: regardless of how good you think the updates are, don’t update the app too frequently (multiple times in a week)  if you have this notification implemented in the app. This becomes annoying and users start reporting it in reviews. This happened when I had to release multiple updates in a week to resolve content infringement complaints submitted by The Verge.

Hope this help.

Update on The Verge app

Converge

UPDATE: I have not received any further response/complaint from The Verge. Considering I have resolved the complaints they submitted to Microsoft in last app update and received their commitment to support the app previously (as explained below), I have published the app again in store. If The Verge has any feedback or want to discuss anything, I welcome them to reach out to me. You have my commitment to work with you to make this app serve your readers in a way which works for everyone.

The Verge is one of my favorite tech news sites. There are always controversies around who likes which product or tech company but overall, The Verge seems on point most of the time. I read that site so much that I created an app for myself as their mobile site isn’t good. Later when I shared the app in The Verge reader community, a lot of folks liked the app and asked if I could add more features. Now, adding features and making an app worth using takes some serious efforts and commitment.

One thing I learned by watching apps come and go is that one shouldn’t develop an unofficial app unless the service has public APIs or you are partnering with the team who owns the service.

  • First, building a brand and service which people want to use takes your sweat and blood. As a 3rd party developer, you shouldn’t use someone else’s work to succeed.
  • Second, even if you don’t agree with the first point, you can, and you will, be forced to remove the app and your efforts will be wasted.

But if you are partnering, that is really a worthy gig. I absolutely loved working with the amazing team at Khan Academy to develop apps for Windows 8 and Windows Phone

Considering these points, I reached out to the Vox Media (owner of The Verge) engineering team to see if they wanted to partner. They loved the app but after their internal discussion, they chose not to collaborate in any way. Their ask was to not use The Verge branding and they wouldn’t have problem with the app. I was bummed but fully respected their decision and started a thread in The Verge forums to get ideas on a new name.

In this same thread, Nilay confirmed that they love the app and want me to continue improving it. Not only that, this was repeated in their weekly verge cast video. As long as the app didn’t use their branding, they would fully support it. This thread has been removed from the forums now. In fact, The Verge has been removing or locking threads which discuss this app in their forums.

With that confirmation, I started adding features in the app. It took hundreds of hours from my weekends and nights but the results were very satisfying, just look at the reviews posted by users. With the most recent update which includes support for adding comments, app is at par with The Verge’s official apps on other platforms.

Yesterday, I received email from Microsoft Store team that The Verge has submitted content infringement complaint against the app for using their name and logo. This is something I had agreed to fix but didn’t change yet so I unpublished the app from store and updated the app to not use there branding. It seems the feature to include app signature while commenting also caused some frustration for the verge staff but they never reached out to me to resolve that or asked to remove the feature.

Once their reported issues were fixed, I reached out to Vox Media lawyer to make sure they are ok with the fixes I have made. They were happy to know that I had removed their branding but, surprisingly, also mentioned that they don’t want this app to distribute their copyrighted content. I explained that this app is like any other thousands of news reader apps which allow users to read content from sites. I am not taking their content and re-publishing or distributing it. They don’t seem to agree with it even though they mentioned that an app like Feedly is ok. How is my app different from Feddly, they don’t mention that. Why are they doing this, I don’t know. But agreeing to supporting the app and now pulling that support is beyond me. It has definitely caused a lot of stress for me.

I have again reached out to their lawyer as well as other folks in their leadership team. I am hoping to hear from them so that this can be resolved and app can be made available in store again. Technically, I have fixed the issues they complained about and I am free to publish the app until they submit another complaint. There is no point in cat and mouse game though so I want to resolve this issue before making the app available again.

By the way, the update with new name and logo is already visible in the store. You can check it out using the deep link as app is not available in search results or lists.

Feel free to chime-in in comments or on twitter if you have feedback or want to share your support.

The Verge app for Windows Phone

If you are part of the technology world, there is a good chance you read The Verge for getting your fix of news, reviews and their fantastic long reads not only related to technology but also science, art and culture. If want to do that on a Windows Phone though, there are no good options. Until now.

The Verge

Over the last few weekends, I have built a new app for Windows Phone for consuming everything The Verge has to offer. Here are the features it currently supports.

  • Super smooth experience for browsing and reading all the latest news, reviews, features, and exclusives.
  • Read comments posted by The Verge readers.
  • Watch “On The Verge,” “The Vergecast,” “Small Empires,” and more without ever leaving the app.
  • Share news and stories with your friends via Facebook, Twitter, and email.

More features like forums and live tiles are not implemented in the app just yet but I plan to add these over the coming weeks. Feel free to suggest or vote on feature requests on the feedback site. Here is some feedback from the community so far.

Great. The best by far. Great design. Better and more good looking than the android app.

As much as most of The Verge crew don’t like WP, this is a great app to prove how good it can be.

I suddenly feel like we don’t need an official app anymore. Awesome.

Obviously, the app is neither sponsored nor endorsed by The Verge or Vox Media.

Khan Academy on Windows 8: Now Open Source

Khan Academy app launched with Windows 8 last October. A few of us worked together to get the first version developed just in time for Windows 8 launch. Since then, there have been a few minor updates and a big overhaul which was published in store last month. This week is another great milestone: the app is now open source and ready for anyone to contribute.

KhanAcademyInStore

How We Got Here

I developed the initial version with a lot of help from folks over at Khan Academy and an internal Microsoft team. It was really a port of Learning Circle phone app to Windows 8 so naturally, this was developed using C# and XAML.

Later last year, a few more folks from Microsfot and BGC3 stepped up to help and we got some funding. That helped to get Pixel Lab team involved with development and after working on it for a few months,  the next version of the app was published in store last month. This new version was developed in HTML5/JavaScript and this is what we open sourced this week.

Get Involved

If you have experience in developing Windows 8 apps using HTML5 and JavaScript, come join us in making a great Khan Academy app for Windows 8. In addition to HTML5 and JavaScript, having good understanding of WinJS, WinRT, TypeScript and web in general would be great. If you don’t have development experience, you can still help with testing.

Let’s Go!

Can technology help fight corruption?

Corruption is corroding humanity. In his TED talk, The good news on poverty, Bono said that corruption is the biggest challenge in fighting poverty. Think about it though, it’s not only the poverty but every part of life which corruption affects. Growing up in India, I got to experience it at every step of life. The level of corruption there is, it has put India at 94th rank in Corruption Perceptions Index 2012. You may think that’s bad, but there are around 80 countries with worse corruption than India. There have been anti-corruption protests/movements across the globe in recent years but any improvements haven’t been seen yet. In India, corruption has really become part of life. Take this for example. If you need to win a Govt. contract at any level, be it a project worth millions of dollars or a small town street repair, there is a fixed cut for officials who make the decision. There is competition on how much the contractors can bride not how cost-effective or efficient they are. And it’s not only the public sector, same goes for political, police, judiciary, corporate and even the NGOs to some degree.

CPI2012

Image credit: Transparency International

As corruption is a covert activity, it is incredibly hard to measure or track. There is no silver bullet to improve the situation overnight or even in foreseeable future. It will take the global citizens significant time and effort, big or small, to eradicate corruption. Today, we have the tools we didn’t have a few years ago. Across the globe, a large degree of population have smartphones and is part of some kind of social network. The penetration of these tools is going to increase every day. Can we use these tools to create a platform for reporting and measuring corruption with engaging experience? Can we use such a platform to create social awareness in communities around us?

Specifically, I am thinking about an open source app for all major platforms including iOS, Android and Windows Phone to crowd-source corruption activities with some fun. These apps should be built on top of a backend service which provides a cross device experience for users to:

  • Report corruption activities and geo-tag them.
  • Use social networks like Facebook and Twitter to sign in and share.
  • View corruption and transparent/non-corruption activities around them or some location on a map.
  • Gamify the system for user engagement and fun. This is not about making it a game but using game elements in corruption reporting as Four Square uses for location services.

These are really some initial ideas I have and there could be number of other features included. It’s hard to say anything about the success or effectiveness of such a platform. There is a great deal of learning and collaboration required for an effort like this. There are also some issues in this approach.

  • For sure, this wont be effective at all kinds of corruption.
  • You may argue that it may not be fair to the people who are being reported (if including names of corrupt people is allowed).
  • Some people might get agitated if this comes in their way of doing things.
  • The data being reported might not be accurate.

I have been looking around but haven’t come across any project like this. I did come across ipaidabribe.com, a site for people to report bribes in India but I didn’t get the feeling of a modern web app by looking at it, there aren’t any mobile apps available. I also looked in iOS App Store and could only find Bribr, bribery reporting app for Russia.

What’s your opinion? Can a project like this help fight corruption? If at all started, would you contribute?

Learning Circle update is now in Windows Phone Store

Learning circle has been in Windows Phone store for over a year now and it has been very well received with consistent 4+ star rating in many markets. I finally managed to submit an update for the app which cleared store certification today.

Learning Circle Panorama

So, what’s new in this release:

  • Removed ads. When I created the app, I made the decision to host a service to support the app which has costs. I wanted to make sure I don’t end up paying for that out of my pocket as the app is free so ads were really a safety net. For months, cost for hosting the service hasn’t been a problem so I decided to get out of the way of learners and leave them alone with Khan Academy’s legendary content. I’m not a big fan of ads anyways.
  • Updated content library with recent additions at Khan Academy and Channel 9.
  • A few other usability improvements such as multi-select list on downloads page.

If you use learning circle on Windows Phone, you should definitely update to the latest version from store. Let me know if you have any feedback.