Integration testing azure storage – Fluent syntax

Hi,

I recently set about writing a solution that’s heavily reliant on Azure Blob storage. I found my debugging cycle wasn’t nice, I’d spin up the code then spend ages in Azure Storage Explorer to work out what had happened. I also new I’d want some integration tests for the future.

So I did some research set about writing a quick set of helpers to allow me to write clean, quick and simple tests that worked in VS Test explorer.

The result is FluentAzureBlobTesting, this allows me to write lovely declarative statements like this:

           blobClient
                .AssertContainerExists(expectedContainerName)
                .AssertBlobExists(expectedBlobName)
                .AssertBlobDataIs(expectedBlobData)
                .AssertBlobContainsMetaData(expectedMetaDataKey, expectedMetaDataValue)
It also handles the starting, stopping and clearing of the storage emulator so all you have to do is click “Run All” and it handles the rest. *Dependency on Azure 2.3 SDK
        private static CloudStorageAccount account;
        private static CloudBlobClient blobClient;

        [ClassInitialize]
        public static void StartAndCleanStorage(TestContext cont)
        {
            account = CloudStorageAccount.DevelopmentStorageAccount;
            blobClient = account.CreateCloudBlobClient();
            blobClient.StartEmulator();
        }

        [ClassCleanup]
        public static void ShutdownStorage()
        {
            blobClient.StopEmulator();
        }

        [TestInitialize]
        public void CleanAndRestartStorage()
        {
            blobClient.ClearBlobItemsFromEmulator();
        }
Should the test fail the extensions report back the reason for failures and log trace output on success.
FailedTest PassingTestWithTrace
Now my debugging cycle is nice a quick and I’m writing a good set of unit tests as I go.
The source is available on github here, hopefully useful to you!
Thanks to Rory for the starting point here: http://www.neovolve.com/post/2012/01/12/Integration-testing-with-Azure-development-storage.aspx

Capture HTTP traffic with Fiddler on Devices with no Proxy option

Hi,

This is how I ended up writing a custom DNS server to redirect network traffic.. code is here.

I recently got a smart TV and I wanted to see what it was up too, having heard all the stories of them leaking personal information left right and center.

Fiddler is my go to tool for any HTTP inspection, so I went about look for a proxy setting in on the TV … there isn’t one.

To get around this I started pondering and came up with a plan to write my own DNS server in C# which would respond to the TV with the IP address of my machine. On that machine I’d configure Fiddler to run on port 80 and allow remote clients. The upshot of this would be that all traffic from the TV would hit the Fiddler proxy, I would inspect it, then it would get forwarded on to the the internet as normal.

Having a search around I found this nuget package, ARSoft.Tools.Net, from ARSoft which gives me the basics of a DNS server. I then hacked together a bit of code to setup the DNS server to respond as I’d like and bound it up to the correct IPs.

The net result was I could now look at *some* of the traffic. The downside is that a lot of apps would break as the HTTPS connections are being forwarded but my machine, and fiddler, but the machine isn’t listening or forwarding port 443.

I had a couple of attempts at getting around this, you can see TCPForwarderSlim and PortForwardingWrapper, thanks to Bruno Garcia, in the code. This would setup a straight forward on 443 when a DNS request would come through but it wasn’t too reliable. I also playing with using fiddlerCore to listen on a number of ports at the same time but this lead to certificate errors in the apps on the TV as the certs it generated where self signed.

Either way it almost worked, screenshots below, I’ve put the code up on GitHub for anyone who wants to have a play! https://github.com/lawrencegripper/FiddlerDnsForwarder

Go a rough idea of some of the stuff my TV gets up to…

For example: If you want the TV version of BBC IPlayer set you’re UserAgent string to the below and hit http://bbc.co.uk/iplayer

Mozilla/5.0 (DirectFB; Linux armv7l) AppleWebKit/534.26+ (KHTML, like Gecko) Version/5.0 Safari/534.26+ LG Browser/5.00.00(+mouse+SCREEN+TUNER; LGE; 42LS570T-ZB; 04.54.03; 0×00000001;); LG NetCast.TV-2012

fiddlersetup redirecting

 

 

 

How I unwittingly became part of a DDOS attack

I recently stood up a azure VM to test out some work I’ve been doing around creating my own DNS server.

Feeling bold and overconfident I created a machine with a long random name, punched open the firewall for dns and set my custom dns running. Having played with the code till late in the evening I left the box running and went to bed. My machines existence was only known to me, so I just left it there with unfinished code answering dns queries.

The next evening, when I set about to continue my work, I noticed that the server was actually already really busy. I was receiving LOADS of requests.

Looking at the apparent source IP it was from a Russian hosting company. However, as DNS is UDP, spoofing the source IP is possible and therefore it can be used it to amplify DDOS attacks. You can read more on this attach type here, http://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack

So it turns out that overnight my machine had apparently been found by a bot crawling IPs and conscripted into a DNS DDOS attack.

Lesson learned, security through obscurity doesn’t cut it on the internet!

Windows Phone 8.1 – Create a Draw control like Android NavigationDraw

Hi,

This is something I wanted to do with my HypeMix app, the aim was that the user would be able to navigate around the app but, on any page, open the draw to see the currently playing track etc.

Luckily, now Windows Phone 8.1 is in line with WinRT, we have the idea of Frames for Navigation.

Using this I created a FramePage, this is the first page the app navigates to and it contains the Draw Content, Frame (for subsequent pages we navigate to) and some animations for show and hide.

Once my custom FramePage has loaded it sets up the app so all navigation happens inside this page, which contains the draw overlay.

As there are quite a few moving parts, rather than copy and paste LOADS of snippets, I’ve put the whole thing up on GitHub for you to play with.

https://github.com/lawrencegripper/WindowsPhone8–NavigationDrawerExample

Ps. Quick and dirty demo hacked together from the actual app, no lovely UI, just some nice bright colors to demonstrate different panels and pages. In my app I’ve combined this with a pub/sub model and a navigation service to avoid the code behind and make things a bit nicer, left out of this demo to aid simplicity.

Hopefully this is useful to others!

 

 

 

Configure Azure Cache From Cloud Config in PAAS

I’ve recently been working with the Azure cache  client which, by default, is configured by storing the details in your app.config or web.config.

Now this is great if you’re in the traditional world but for Azure Platform as a service solutions Cloud Configuration is key. It allows you to change settings easily across multiple machines without having to jump into web.configs or redeploy etc.

I wrote the little method below to help me to do this as in my testing I was jumping between a couple of different cache’s regularly.

You can then use the CloudConfigurationManager to retreive the info needed then call into the method to start using the cache:

CloudConfigurationManager.GetSetting("CacheDiscoryUrl")
CloudConfigurationManager.GetSetting("Token")

You may want to alter to use something other than the “GetDefaultCache” method but hopefully a good starting point.

[This method was tested in a limited way, double check it makes sense for you before going nuts and putting it into prod!]

Moving existing Win 8.1 App to Win Phone 8.1 with new Universal App project – HypeMix has a go!

This a quick and dirty write up of how I moved my existing Windows 8.1 App to Windows Phone 8.1 in a weekend. Now before we get into that I’m going to be honest and say there is still some work to do around the UI, as playing music on the phone is slightly different to a PC or Tablet. What I’m talking about here is all the rest of the work.

So, I set out on my ambition goal with a simple step. My current solution had 4 Projects, one for the app, one for the API that it uses, one for tests and a second for tests just to be safe. If you want to play with the app you can get it here – http://apps.microsoft.com/windows/en-gb/app/hypemix/f5e81d5d-b972-456c-9f33-655c23488815

Step 1:

Right click on the Win 8.1 Project and Click “Add windows Phone project”

This gives you a “Shared” section, if you’ve worked with linked files before this is along those lines. When you compile each of the Win or WinPhone projects, the files in shared are included.

Step 2:

 

Now that you’ve got this lovely shared section you can move over all the obvious bits that you want to share. To kick this off I moved over my Commands, Exceptions, Services and ViewModels.

(To clarify the app uses Reactive Extensions to create a Publish and Subscribe architecture. When you click a button this binds to a Command on a ViewModel. The Command publishes an Event and these events are subscribed to by one or more services.

For example, Clicking “Play” invokes the “PlayPauseCommand” which in turn publishes a “PlayPause” Event. This event is subscribed to by the “MediaService”, when received the service starts playing the track. It’s also subscribed to by the “NotificationService” which pops up a toast to inform the user about which track is playing)

The first awesome thing I saw here was that my existing Win8.1 app still worked fine, moving the stuff to the shared section hadn’t had any effect on it. Big win, no changes needed to existing app so far!

The second awesome thing I saw here was EVERY BIT OF CODE, not counting the XAML and code behind ~8 files, was shared. That’s right, EVERY ONE OF THE 1.2K LINES OF CODE in the Windows 8.1 Solution could be shared. So far so good, let’s see how the API project gets on.

Step 3:

I mentioned earlier that I have a Windows 8.1 Class Library, which implements the HypeM Api. Now in the new SDK we have Universal App – Portable Class library. This is just a PCL which targets Win8.1 and WinPhone 8.1. So I can take advantage of this I needed to move all my stuff over to the new project type.

 let me know that you can simply retarget your project to Windows Phone 8.1 by going into properties and selecting Windows Phone. As this is a much simpler than using linked files I’ve cross out my original bit and moved to using this!

Bko0vYICQAAo-aD

 

To do this I created a new “Universal App Portable Class Library”.

Then I used project linker to link all the files and add them into the new project type. (I had to download the VSIX file and alter the manifest so that it would work with VS2013 but once I’d done that it was all good, VSIX below)

After re-targeting the library I hit my first snag. When I happily hit F6, expecting the same level of success, my heart sunk as I saw errors. “Here we go” I thought, I’m going to have to refactor loads of code.

Once I’d talked myself back from the edge and actually looked at the errors I found there was only one culprit.

This was the credentials picker, which isn’t currently in the shared WinRT Core.

using Windows.Security.Credentials.UI;
CredentialPickerOptions opts = new CredentialPickerOptions();

This is the awesome bit, apart from that everything else just worked! That’s another 1.4k+ lines of shared code.

 

Once again I jumped back to the Windows 8.1 app and pressed F5 and was happy to see the existing app still working well, with its new projects and shared core.

 

Next – Build the Windows Phone 8.1 Project

 

I switched over and tried to build the Windows Phone 8.1 Project, the errors quickly stacked up and I had another “Oh this is going to be tough” moment.

Actually it wasn’t, in step 2 I moved some stuff to the shared project. The odd one here is that any references required for this code need to be referenced in both the Windows Phone and Windows 8.1 project. This is because, as I mentioned, the Shared bit isn’t actually a project, it’s a bunch of files and as such it doesn’t have references.

Now normally I’d just jump on nuget to get hold of these references, which is where the second bit of trouble struck …

 

I’d used the Prism for Store Apps package for a number of things in the Win8.1 app. When I tried to add it from Nuget I got some nasty red telling me it wasn’t supported by this project type.

I thought to myself, now WinRT is basically shared I wonder if I can use the Store version. Sure enough add it manually and you’re golden!

 

The second roadblock I hit was around App Insights, I couldn’t get this to play nice with Phone 8.1. However as I’m using a nice Pub/Sub model for events I can just leave this be for the time being while they’re SDK gets updated.

I will look to switch out the “AppInsightService” below with a WinPhone friendly one once it becomes available.

 

public class AppInsightService
   {
       public AppInsightService()
       {
           EventPubSub.Subscribe<CurrentPlaylistUpdate>(x =>
           {
               var properties = new Dictionary<stringobject>() 
{ { "artist", x.CurrentPlayingTrack.artist }, { "track", x.CurrentPlayingTrack.title }, { "id", x.CurrentPlayingTrack.itemid } };
               ClientAnalyticsChannel.Default.LogEvent("Updated Song", properties);
           });

 

So I started a little platform specific folder in my Win 8 Project. Now containing app insights and ApiAuth, which uses the Credentials UI mentioned earlier.

 

 

Then I had some fun sorting out some more references from the Windows Phone Side, I used Unity for my decency dependency injection and needed to get this up and running in the Phone Project. Again I saw issues with Nuget not playing nice with the new Project type.

So I jumped into my “packages” file and found the NetCore45 version that was currently used in the main Windows 8.1 App.

 

Adding this manually and again I was all good!

 

Unity and Next set of errors….

Errors1

 

In my Windows 8.1 App I have a static reference to the Unity container in the App.xaml.cs. Rightly or wrongly I configure the container in the App.Xaml.Cs and use this to keep a static reference to it while the app is running.

I could see my Windows 8.1 Project building fine but not my Windows Phone project. This was because, as the Win8.1 builds, it pulls in the shared files from the new Shared area and all the references to “App.Container” are present in its App.Xaml.Cs.

When I build the Windows Phone 8.1 the errors appear as the Phone project doesn’t have that container setup in its app.xaml.cs. This was a good lesson, be careful when using static stuff which hangs off the App object in your shared section, as you’ll have different versions in each project!

Actually, in this case, it’s a nice thing as it makes perfect sense to have different Unity config for each app. For example, one of the items that unity resolves in my ApiAuth.cs – which we’ve had to move into platform specific folder in the Windows 8 Project as it couldn’t be shared. So in each App.Xaml.Cs I can have different unity config which provides a Platform specific version of that class.

Navigation

 

Next tricksy one is navigation, as you can see from these error messages I’m using a static reference to the rootframe to move around. This isn’t going to do the trick, as we’re going to have different types depending on which version we’re in because the Pages don’t sit in the Shared Section.

 

 

Errors

My first thought here was to use Unity, as I’ve already got the way paved for me. I was going to implement a quick Interface and use the unity configuration to inject a WinPhone of Win8.1 Navigation handler in each of the projects. That way I drop this nasty stuff and I can map a navigate event to a different page based on the platform etc.

When I thought about it a bit more though I decided to go with leveraging the existing publish subscribe model, with services, I’m using in the app.

I created a “NavigationEvent” with “param” and “PageName” then a service which subscribes to these events and kicks off the navigation events.

 

namespace HyperM8.Shared.PubSub
{
    public class NavigationEvent
    {
        public string PageName { getset; }
        public string ParamsString { getset; }
    }
}

This led to changing around 5 or 6 of my commands to using this model.

var navEvent = new NavigationEvent(){ PageName = "MoreItemsPage", ParamsString = string.Format("{0},{1}", url, blog.sitename)};
   EventPubSub.Publish<NavigationEvent>(navEvent);

 

VS the existing direct reference to the rootFrame to move around.

App.RootFrame.Navigate(typeof(MoreItemsPage), string.Format("{0},{1}",song.PlaylistUrl, song.SongCategory));

 

(What I also realised, during this process, was that I should be injecting my pubsub bus “EventPubSub” as an Interface so I can mock it out and test with something like MOQ. So a side effect of some of this refactoring was that I ended up breaking some of my tests.)

Next step was to implement the navigation service in the platform specific folder on the Windows 8.1 project.

I’m using a nice bit of functionality in Unity here called AllClasses. This allowed me to interrogate the classes loaded in the application and find a type that matched the “PageName” passed into the “NavigationEvent” then navigate to that page.

 

public NavigationService()
{
    EventPubSub.Subscribe<NavigationEvent>(navEvent =>
    {
        var type = AllClasses.FromApplication().Where(c => c.Name == navEvent.PageName).FirstOrDefault();
        if (type == null)
        {
            Debug.WriteLine("Failed to navigate to {0}, type not found", navEvent.PageName);
            return;
        }
        App.RootFrame.Navigate(type, navEvent.ParamsString);
    });
}

(@robgarfoot has pointed out that this is quite a bit of overhead, going through every class in ever assembly in the app each time a nav event is fired. So far I haven’t seen a huge hit from this but I do agree this is a bit smelly. When I have a bit more time I’m going to look at using Unity to make this a once time mapping, rather than something that happens every time the app navigates)

So with this now in place and all the commands updated, my project builds and my Windows 8.1 app is still working nicely. What I then realised was that I could put this “NavigationService” into my Shared section and as long as I named the pages correctly it would all work and also be shared.

Summary

 

I’ve shared a HUGE AMOUNT OF CODE and I’m way ahead where I would have been if I’d tried this before this updated. The only bits where I saw issues was with Navigation, Nuget Packages, References and Credentials Picks… none of which took me more than 15mins each to get around…

 

In short…. THIS IS AWESOME!!

 

My project structure is as follows, unloaded have either been killed by my changes or legacy.

 

 

Next up was migrating some of the XAML for the HubPage. Certainly there are more differences here. I got a massive helping hand from the new approach to Universal apps and project. I took a chance and moved my Resource Dictionary, containing styles and Data Templates, into the Shared Project.

 

 

Next I updated the app.xaml in both Win8.1 and WinPhone to pick up these styles and merge them into the dictionary. (Yes, there is a file called MupetStyles. No, I don’t have a clue why I called it that either).

 

<Application x:Class="HypeM8.App" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:local="using:HypeM8">
    <Application.Resources>
        <ResourceDictionary>
            <ResourceDictionary.MergedDictionaries>
                <ResourceDictionary Source="StandardStyles.xaml" />
                <ResourceDictionary Source="MupetStyles.xaml" />
            </ResourceDictionary.MergedDictionaries>
        </ResourceDictionary>
    </Application.Resources>
</Application>

 

To my absolute delight, this worked and my styles came across. Emboldened by this I picked up the body of my existing Windows 8.1 HubPage.xaml and Ctrl-C, Ctrl-V’d it into a page on the phone. With the exception of about 15 errors relating to slight differences in the XAML, THIS ALSO WORKED! I EVEN GOT MY NICE DESIGN TIME EXPEREINCE!

Just to emphasize this, I’d shared all my converters, styles and most of the page XAML between the projects. I still am completely blown away by this!

 

I was now at the stage where my app was building, had a working design time view and also nearly everything was shared (API, Converters, Services, Styles, ViewModels…)

Next – F5 onto the Emulator and see if it works for real

 

So by this point, if you’re anything like me, you’re thinking it’s all going to go wrong. And it did for a little bit, I came up against this cryptic little error while working with a message dialog.

 

After about 20mins of trying out different stuff I worked out that this actually means “MessageDialog” with more than two buttons isn’t supported on the phone 8.1. With that all ironed out I eagerly awaiting the next F5.

There I was, my app ported over with the first page up and running. Sure I needed to do some work on tweaking styling, porting other xaml pages and implementing a login screen but apart from that I was there. I’d shared nearly 2k lines of code and got to this point inside 2 days.

Hopefully this was a good guide for others trying to do the same, keep an eye out for HypeMix in the Windows Phone 8.1 Store once I’ve tweaked the styling and finished the rest of the pages. Depending on how I get on I may do a follow up post.

 

 

Windows Store App – Infinite Scrolling list with Xaml

I recently came across a great interface and behavior in windows 8.1. If you implement ISupportIncrementalLoading on your bound property then bind up to a gridview you’ll get infinite scrolling for free.

Below is an example from my HypeMix app where I pull in more tracks and add more to that list as the user scrolls, hopefully useful to others.

Tagged , ,

Azure Store – Free MongoDB, New Relic, SendGrid and more..

Hi All,

I set about to put together a new app and challenged myself to do the whole architecture on azure without spending a penny. The only caveat I set was that I should be able to scale if everything goes really well.

So the basic components I’m using are:

  • MongoDB – 500mb db offered on Azure Store by MongoLabs (having not played with mongo wanted to give it a go, you could also use SQL Azure free db or others)
  • SendGrid – 20k emails a month
  • New Relic – Free edition
  • Azure Websites – Free site

I was amazed hat you can get so much for free, I can basically stand up a monitored, scalable architecture with no outlay and then look at scaling as needed – based on the success of my services.

This is a great proposition for any entrepreneurial devs out there, grab the tooling and put your app together with no running costs while it gets off the ground!

Have a look around here:

http://www.windowsazure.com/en-us/gallery/store/

mongo

Clean up unused Services from Azure Portal with PowerShell

So I’ve recently been using lots of VM’s and I end up getting through them fairly quickly – deleting and creating them regularly.

What I ended up with was loads of ghost services in the portal that were, once upon a time, associated with a VM in the distant past.

Given PowerShell is awesome, I set about automating the removal of these using the azure cmdlts. The trick is that I don’t want to remove services or VM’s that are currently “Stopped Deallocated” only those ghost services.

The script has three stages:

  1. Find all VM’s and their associated services.
  2. Find all the services without active deployments.
  3. Identify if any of these empty services have VM’s associated with them.
  4. Perform remove action on these services.

Start and Stop Azure VM’s (and more) in Parallel from Powershell

So I’ve got a demo that I’ve put together that uses a LOAD of Azure VM’s to demonstrate scripting against them.

To make sure I don’t get charged for them while I’m not using them I’ve got the following powershell to start and stop them:

Get-AzureVM | Stop-AzureVM –Force

Get-AzureVM | Start-AzureVM

I notice that this took a longer than I was expecting and so I started to do some digging. It turn out the PowerShell pipeline is executing these API requests in a synchronous way. Firing off the first request, waiting for the response and then, when completed, moving on to the next. This wouldn’t be a problem if I didn’t have loads of VM’s!

So I wrote the following, it uses PowerShell Jobs (think Task<T> if you’re a C# person), to kick these off as background jobs that run in parallel. (nb. You have to pass the variables in as arguments to the script block)

 

Now we get this lovely output. These all run as individual jobs and complete asynchronously, we then wait for all of them (think WaitAll() for C#). Now I can start all of my machines in the demo really quickly!

This is really useful in other scenarios too, think of any time you’re waiting on azure management API and the actions could be done simultaneously.

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command

–     —-            ————-   —–         ———–     ——–             ——-

22     Job22           BackgroundJob   Running       True            localhost            …

24     Job24           BackgroundJob   Running       True            localhost            …

26     Job26           BackgroundJob   Running       True            localhost            …

28     Job28           BackgroundJob   Running       True            localhost            …

30     Job30           BackgroundJob   Running       True            localhost            …

RunspaceId           : 88465362-c8ce-45de-866d-9ced85cc3e40

OperationDescription : Start-AzureVM

OperationId          : 60712d20-311b-6cd5-b0df-9a750f024d0c

OperationStatus      : Succeeded

RunspaceId           : 3a9e93cf-7d16-4a2c-aeb7-4493e8ec8abb

OperationDescription : Start-AzureVM

OperationId          : fb494c0c-3728-62d4-b288-b8d391ad32d7

OperationStatus      : Succeeded

RunspaceId           : c1144e74-ba7c-4aa3-bcbb-aa9f2bd72fdf

OperationDescription : Start-AzureVM

OperationId          : 194254da-be36-6c7d-a147-67a980816c8e

OperationStatus      : Succeeded

RunspaceId           : 44af74a2-fc11-4e9b-8158-d9d84dd49d9e

OperationDescription : Start-AzureVM

OperationId          : 86059d5d-22ee-6a55-871d-28bcf52241a2

OperationStatus      : Succeeded

RunspaceId           : a44b51e9-11cc-4d31-875e-5e0fd16b935a

OperationDescription : Start-AzureVM

OperationId          : d4e2fcce-b270-6d6e-ad0c-8322e2c4df23

OperationStatus      : Succeeded

Follow

Get every new post delivered to your Inbox.

Join 282 other followers