Technologies

  • ASP.Net MVC
  • ASP.Net API
  • Blob Storage
  • NoSQL (DocumentDB), SQL
  • Azure Service Bus
  • Cloud Services
  • D3
  • Scheduler

PowerBi

(Bonus)

Using PowerBi desktop's DocumentDb adapter, I connected the scraped football data in

Swagger

(Bonus)

Using swagger(Swashbuckle) i was added a swagger endpoint to my API and generated XML documentation 

API

Big Football Data

Concept

Gathering data from various sources and displaying the data in a meaningful way.

The use of computer supported, interactive, visual representation of abstract data to amplify cognition

Technologies

  • ASP.Net MVC
  • ASP.Net API
  • Blob Storage
  • NoSQL (DocumentDB), SQL
  • Azure Service Bus
  • Cloud Services
  • D3
  • Scheduler

PAAS

Platform as a service

Step 1: Getting the infomation

The first stage is to gather our data and store it.

It is important during this stage that we don't modify the data in anyway, we want to be able to refer back to the source to scrape additional information.

http://www.correct-scores.co.uk/form_Premiership.html

Data Feed

Extractor

The extractor is a cloud service that will identify the feed type, retrieve the raw data, store in Blob format and then place a message on the service bus.

Blob Storage

Why?

 Azure Blob storage offers a pay as you go solution with lots of metrics out of the box so you can monitor the performance of your data

What?

Azure Blob storage is a service to store stuff (Objects) 

Step 2: Scraping the data

Now that we have the raw data we can start to pull information out, we will also need to look at how we are going to store this data.

Scrapers

The Scrapers are a cloud services similar to the extractor they will pull information out of the raw data and save it into DocumentDb

DocumentDb

What?

Simply put DocumentDB is a NoSQL database specifically the managed Json Document kind.

“Database Service” means Azure manages the infrastructure and you pay for the data out the tap

DocumentDb

Why?

  • Dynamic schema, this means that as our data changes so does our schema lots.
  • Horizontal scaling, as our business needs grow so does our database.

 

Step 3: Querying and Displaying the data

Now we have collected and stored the information we can start working with it.

Ephemeral APIs mean that the endpoints and payloads should be able to be terminated and created with ease and flexibility

API

We wrap our services behind an API then give our users access to multiple lightweight endpoints that are designed to deliver specific data, this allows us to have lots of services tucked behind one API that has the capability to deliver a wide range of functionality

This pattern of a separated architecture is a popular technique sites like Flickr, Facebook, Twitter, Photobucket, LinkedIn and Sony take advantage of information hiding.

 

This allows functionality on any platform that can take advantage of HTTP (Mobile, Gaming, Web) improving the UX

API

ASP MVC

Is the process of enhancing user satisfaction by improving the usability, accessibility, and pleasure provided in the interaction between the user and the product.

The UX is a ASP Web application that displays data using D3, it uses custom roles to elevate users into admins. Admins can interact with the API to create, update and perform scrapes on new feeds.

 

What Is Data Conflation

To conflate, mix together or blend.

Data conflation is essentially taking separate pieces of data and combining them to make a different picture of the data.

Data visualization

The use of computer supported, interactive, visual representation of abstract data to amplify cognition.

 

Data Driven Documents

Using D3 to visualize and improve the end product of our data

A

(Data)

B

(Visual)

Stage 1: Getting the infomation

function renderFormBar(form) {
    var svg, square, startPos = 50;
    for (var i = 0; i < form.length; i++) {
        form[i].Align = startPos;
        startPos += 50;
    };
    svg = d3.select(".form-bar").append("svg").attr("width", startPos);
    
    //Code Appended    

    d3.json("", function() {
        square = svg.selectAll("rect")
            .data(form).enter()
            .append("rect")
                .attr("width", 40)
                .attr("height", 40)
                .style("fill", function (d) { return formColour(d) })
                .on('mouseover', tip.show)
                .on('mouseout', tip.hide)
                .attr("x", function (d) { return d.Align });
    });
}
function formColour(game) {
    if (game.Scored < game.Conceded) {
        return "Red";
    }
    if (game.Scored > game.Conceded) {
        return "Green";
    }
    if (game.Scored === game.Conceded) {
        return "Orange";
    }//Code appended
}

Demo

Step 4: Automation

Now that we have a our services we can start to look at automating the whole process, to do this we need to have knowledge of our data sources.

 

Super Sunday, Monday Night Football

Most of use are aware these are the biggest days in the week for the EPL.

All the games are finished by 9 and the stat-sites are updated by 10, we can identify this pattern and start to fine tune our application...

 

Scheduler

Azure Scheduler lets you create jobs in the cloud that reliably invoke services inside and outside of Azure—such as calling HTTP/S endpoints or posting messages to Azure Storage queues, Azure Service Bus queues, or Azure Service Bus topics. You can choose to run jobs straight away, on a recurring schedule or at some point in the future.

With this knowledge we can create a scheduler that will generate a brokered message that we can stick on our service bus at given time, we can even make this a recursive process

First you create a new scheduler collection and configure a new scheduled job to run at you service bus at a desired time...

We can apply other scaling rules based on metrics like CPU usage, network traffic, memory usage...

Messaging

Azure Service Bus is a cloud-based messaging system that allows you to connect applications, we use it to connect our cloud services to the outside world.

We send scrape and extraction requests in the form of brokered messages

Queues

public override bool OnStart()
{
    _queueClient = QueueClient.CreateFromConnectionString(ConnectionString, QueueName);
    var namespaceManager = NamespaceManager.CreateFromConnectionString(ConnectionString);
    if (!namespaceManager.QueueExists(QueueName))
        namespaceManager.CreateQueue(QueueName);
    //...

    return base.OnStart();
}
public override void Run()
{
    //..
    while (true)
    {
        var message = _queueClient.Receive();
        if (message != null)
        {
            try
            {
                Trace.TraceInformation("Processing Service Bus message");
                ReceivedMessage(message);
                message.Complete();
            }
            catch (MessagingException ex)
            {
                if (!ex.IsTransient)
                {
                  //...

Topics

    public void SetUp()
    {
        //Code omitted
 
        if (!namespaceManager.SubscriptionExists(TopicName, SubscriptionName))
            namespaceManager.CreateSubscription(TopicName, SubscriptionName, new FalseFilter());

        _subscriptionClient = factory.CreateSubscriptionClient(TopicName, SubscriptionName);
        
        //...

        _subscriptionClient.AddRule(new RuleDescription
        {
            Name = "BasicListenRule",
            Filter = new SqlFilter("ScraperId = 'Statbunker'"),
        });
        _subscriptionClient.AddRule(new RuleDescription
        {
            Name = "HrefListenRule",
            Filter = new SqlFilter("ScraperId = 'StatbunkerHref'"),
        });

        //...
    }

Publish, Subscribe with SQL filtering rules

Demo

Problems

Azure is constantly changing, this is both good and bad, most of the time it is to add new feature but sometimes it can depreciate existing work.

NServiceBus vs Azure Service Bus

No new data

Feeds being removed

Azure credit limit

Monolith to Microservices

TSVS to Github

Scrum to Kanban

Mapping the API

The Architecture

The Architecture

The Architecture

The Architecture

The Architecture

deck

By jwcnewton

Made with Slides.com

deck

  • 147

More from jwcnewton