Business Intelligence (BI), which like government intelligence sounds faintly like an oxymoron, has been around a long time. The earliest reference to business intelligence appeared in 1865, according to a Wikipedia article. In more modern times, the term started to appear at IBM in the late 1950s but Gartner is quoted as saying BI didn't gain traction in the corporate world until the 1990s. So it appears to be a term coined at the end of the Civil War that then moved into common usage with Decision Support Systems for data-based tactics and strategies, which were developed from 1965 to 1985. Any way you look at it BI is not new technology.
Fast forwarding to the present where massive amounts of data are being gathered from the huge surge in IoT devices and processed by increasingly sophisticated Artificial Intelligence software, the new term is Business Analytics. This is not your grandfather's punch cards being fed into water-cooled mainframe computers. It's not even business analysts working with Lotus 123 on MS-DOS PCs in 1985. Business Analytics is a whole new ballgame and if you want to play in the game, you may need to upgrade your skillset.
How Much Data Is Too Much?
With IoT and other edge computing systems collecting billions and billions of event and process data, some industry watchers are predicting a "Data-apocalypse" where there will be more information than cloud storage can handle or AI can analyze. But a more optimistic view is that the vast amounts of data being gathered globally present an unprecedented opportunity to give business decision makers a wealth of facts organized in applications for developing tactics and strategies. How are products really doing in the marketplace? Are the processes that create those products as efficient and safe as they could be? Who actually wants to buy what you are selling? Business analytics offers the potential to greatly reduce the guesswork that amounted to little more than a coin flip in old time decision making.
From Theory to Applications
Of course, this is not going to magically appear on business users desktops. Applications need to be built to meet the specific business analytics needs of individual companies, departments and groups. There are tools developers can use to get those jobs done, including Microsoft Data Platform technologies such as SQL Server, Azure SQL Server, and even good old Power BI.
There is news this month on updates to some of the Microsoft technologies specifically to help handle big data.
On Nov. 5, Azure Data Studio was released which is designed to provide multi-database, cross-platform desktop environment for data professionals using the family of on-premises and cloud data platforms on Windows, MacOS, and Linux. More information about it is available on this GitHub page.
The recently released SQL Server 2019 (15.x), SQL Server Big data Clusters is designed to "allow you to deploy scalable clusters of SQL Server, Spark, and HDFS containers running on Kubernetes," as explained in a Microsoft post What are SQL Server Big Data Clusters? "These components are running side by side to enable you to read, write, and process big data from Transact-SQL or Spark, allowing you to easily combine and analyze your high-value relational data with high-volume big data. SQL Server Big Data Clusters provide flexibility in how you interact with your big data ... You can query external data sources, store big data in HDFS managed by SQL Server, or query data from multiple external data sources through the cluster. You can then use the data for AI, machine learning, and other analysis tasks."
That Microsoft post provides scenarios for making use of big data for analysis tasks utilizing AI and machine learning.
Get Up to Speed with an Expert
If you are interested in enhancing your Business Analytics skill set including gaining greater knowledge of the Microsoft Data Platform technologies such as SQL Server, Azure SQL Server, and more, there's a workshop for you this month at Visual Studio Live! From Business Intelligence to Business Analytics with the Microsoft Data Platform is an all-day workshop that is part of Visual Studio Live! in Orlando, Florida, Nov. 17 -22. It is being taught by big data expert Jen Stirrup, who is a Microsoft Data Platform MVP.
This workshop will cover real-life scenarios with takeaways that you can apply as soon as you go back to your company.
Here are the topics included:
- Introduction to Analytics with the Microsoft Data Platform
- Essential Business Statistics for Analytics Success: the important statistics that business users use often in business spheres, such as marketing and strategy.
- Business Analytics for your CEO – what information does your CEO really care about, and how can you produce the analytics that she really wants? In this session, we will go through common calculations and discuss how these can be used for business strategy, along with their interpretation.
- Analytics for Marketing – what numbers do they need, why, and what do they say? In this session, we will look at common marketing scenarios for analytics, and how they can be implemented with the Microsoft Data Platform.
- Analytics for Sales – what numbers do they need on a sales dashboard, why, and what do they say? In this session, we will look at common sales scenarios for analytics such as forecasting and 'what if' scenarios, and how they can be implemented with the Microsoft Data Platform.
- Analytics with Python – When you really have difficult data to crunch, Python is your secret Power tool.
- Business Analytics with Big Data – let's look at big data sources and how we can do big data analytics with tools in Microsoft's Data Platform.
You can find out more about Visual Studio Live! in Orlando here.
Posted by Richard Seeley on 11/20/20190 comments
Any developer can get into building Artificial Intelligence apps with Microsoft Cognitive Services.
Just as in the pioneering days of ecommerce, you could build an online retail store app by dropping in a shopping cart web service, Microsoft wants to empower you to create AI apps by adding pre-built cognitive services.
Suddenly your next app can include machine learning capabilities for speech recognition, text analysis, image recognition, and decision support. The best part is you don't have to be an AI guru or a data scientist to build that app. That is the democratizion of AI that Microsoft is hyping.
As the company says in a recent post explaining Cognitive Services and machine learning: "You don't need special machine learning or data science knowledge to use these services."
If you know how to work with REST and APIs you're good to go as a Microsoft blog defining cognitive services states:
"Cognitive Services are a set of machine learning algorithms that Microsoft has developed to solve problems in the field of Artificial Intelligence (AI). The goal of Cognitive Services is to democratize AI by packaging it into discrete components that are easy for developers to use in their own apps. Web and Universal Windows Platform developers can consume these algorithms through standard REST calls over the Internet to the Cognitive Services APIs."
Applications: Meet LUIS
All this data science abstraction is interesting but how does it work in a real world app? Microsoft has examples, such as its cognitive service called LUIS explained here:
Language Understanding (LUIS) is a cloud-based API service that applies custom machine-learning intelligence to a user's conversational, natural language text to predict overall meaning, and pull out relevant, detailed information.
A client application for LUIS is any conversational application that communicates with a user in natural language to complete a task. Examples of client applications include social media apps, chat bots, and speech-enabled desktop applications.
Welcome to Cognitive Computing
Apps like a LUIS-powered chat bot fall into the brave new world of cognitive computing, which Wikipedia defines in this article.
"Cognitive computing (CC) describes technology platforms that, broadly speaking, are based on the scientific disciplines of artificial intelligence and signal processing. These platforms encompass machine learning, reasoning, natural language processing, speech recognition and vision (object recognition), human–computer interaction, dialog and narrative generation, among other technologies."
As the article goes on to explain, cognitive computing is basically AI technologies being used to act like a human in interactions with actual humans. LUIS will help you create a chat bot to talk to visitors to your website or online retail store.
Microsoft Content Moderator is a service that acts like a human editor checking to make sure content posted to your site does not contain offensive material. The company envisions lots of jobs for this cognitive service including:
- Online marketplaces that moderate product catalogs and other user-generated content.
- Gaming companies that moderate user-generated game artifacts and chat rooms.
- Social messaging platforms that moderate images, text, and videos added by their users.
- Enterprise media companies that implement centralized moderation for their content.
- K-12 education solution providers filtering out content that is inappropriate for students and educators.
Considering all the news and controversy surrounding online postings, working with this technology may be a hot zone.
Cognitive Service is different from machine learning
But before you start featuring yourself as a machine learning aficionado, Microsoft hastens to add that there are differences between ML and its cognitive services offering.
"A Cognitive Service provides a trained model for you," the company states. "This brings data and an algorithm together, available from a REST API(s) or SDK. You can implement this service within minutes, depending on your scenario. A Cognitive Service provides answers to general problems such as key phrases in text or item identification in images."
If you're going to get into ML, you are going to need to up your AI and data science skill levels.
As Microsoft differentiates it: "Machine learning is a process that generally requires a longer period of time to implement successfully. This time is spent on data collection, cleaning, transformation, algorithm selection, model training, and deployment to get to the same level of functionality provided by a Cognitive Service. With machine learning, it is possible to provide answers to highly specialized and/or specific problems. Machine learning problems require familiarity with the specific subject matter and data of the problem under consideration, as well as expertise in data science."
Okay, so it's back to the classroom if you want to learn to do all that stuff.
Learn AI Application Development
At Visual Studio Live in Orlando FL, Nov. 17-22, 2019, there is a track where you can Learn AI Application Development.
"Artificial Intelligence is changing how you build apps," states the course description. "Technologies like advanced machine learning, natural language processing, and more, mean you can build truly 'smart apps'. With today's accelerated development cycles, you need to be ready to build the next generation of self-learning applications. In this track, you'll find talks on interacting with intelligent agents like Alexa and Google Assistant, Microsoft's Cognitive Services, Azure Machine Learning, Databricks and DevOps for AI."
Find out about all the tracks at Visual Studio Live! in Orlando here.
Posted by Richard Seeley on 10/16/20190 comments
If you are developing business to consumer applications but you're not an expert in configuring secure identity management, Microsoft says it can provide everything you need with its Azure Active Directory (AD) Business to Consumer (B2C) identity offering.
AD B2C will do all the heavy lifting when it comes to things like third-party authentication, Single Sign On (SSO) and even protection against Denial of Service and brute force password attacks, the Redmond-based software company says.
To show how easy it is for developers to use, Microsoft produced a YouTube video titled What is Azure Active Directory B2C? In the video, posted this month, Adam Stoffel, Senior Product Manager Azure AD, makes a bold statement: "Microsoft will act as the secure front door to any of these applications and we'll worry about the safety and scalability of the authentication platform."
"B2C can also centralize collection of user profiles and preference information and capture detailed analytics information about behavior and sign up conversion," Stoffel explains. "By serving as central authentication authority for all your applications, B2C provides you with a way to build a single sign on for any API, web or mobile application. We will handle things like denial of service, password spray and brute force attacks, so you can focus on your core business and stay out of the identity business."
Microsoft is clearly assuming most B2C developers want to stay out of the identity business.
The video includes a demo of Azure AD B2C doing its stuff for an online grocery store. It shows how signing up and signing on looks to the user. It also shows how the user profiles can be customized. Say you are allergic to peanuts. The application will flag any product with ingredients that might impact your allergy.
Read All About It
For those developers who actually want to read documentation rather than watch how-to videos. Microsoft has some detail instructions on using Azure AD B2C.
There is actually a written version of What is Azure Active Directory B2C? that is a little more technical and freer of marketing hype. The authors, a committee of Microsoft techies, aim this documentation at all levels of expertise.
"If you're an application developer with or without identity expertise, you might choose to define common identity user flows using the Azure portal," they explain. "If you are an identity professional, systems integrator, consultant, or on an in-house identity team, are comfortable with OpenID Connect flows, and understand identity providers and claims-based authentication, you might choose XML-based custom policies."
This doc covers:
- Protocols and tokens
- Tenants and applications
- User journeys
- Identity providers
- Page customization
- Developer resources
This doc takes about eight minutes to read if you don't move your lips.
There are also links in it to more documentation and tutorials.
Up Close and Personal Training
If your preference runs to hands-on learning rather than watching videos or reading copious amounts of documentation, VS Live! in Chicago, Oct. 6 – 10, offers Reach Any User on Any Platform with Azure AD B2C. Nick Pinheiro, Microsoft Cloud & Software Architect, will be the instructor for this session aimed at the introductory to Intermediate level.
"In this session," the description says, "you will learn how to allow users to login to your web or mobile apps with their social and consumer identities or email address with Azure AD B2C. Technologies include: Azure AD B2C, Azure App Service, API Apps, Xamarin and more."
It will cover:
- Enabling and configuring Azure AD's Business to Consumer (B2C) identity offering
- Using the Microsoft Graph API to access the user data in Azure AD B2C
- Integrating your existing app identity with Azure AD B2C
Posted by Richard Seeley on 09/19/20190 comments
Full stack web developers are in great demand with attractive salaries being dangled from job search sites.
Is this job for you?
If you've heard about this employment category and are intrigued by the possibilities, you still might want to look before you leap.
To start with: What exactly are we talking about when we talk about full stack web development?
It is a little like having access to all the rooms in the mansion. As a full stack web developer, you know how to work in the front rooms and the back rooms and you have access to all the connecting hallways and rooms in between.
The upcoming VSLive! in Chicago has a learning experience that is a full track on the full stack. Here's how it's described:
Full Stack Web Development
Topics in this track include:
- ASP.NET Core intro and deep dive
- Getting started with Razor Pages
- Angular 101
- Managing async data in Angular
- Designing advanced Angular components
- Advanced Azure App Services
- Advanced TypeScript
So there you have the components of what you need to know to get started in the full stack world. There's a lot going on in the Microsoft world with ASP.NET Core as well as Azure, Razor, Angular and Typescript. Are you up to date on these core technologies? Here is the latest news.
Can You Say Azure?
Since Oct. 2008 when Microsoft announced its cloud computing service, Azure has been the most mispronounced product name in history. Do a search on how to say Azure and many variations appear. The American pronunciation is said to be a.zhr. But a lot of computer industry analysts seem bent on outdoing each other in coming up with different ways to say it. Originally, before Microsoft presumably took the name from an RGB color wheel, azure was a variation of the clear blue sky. Maybe that sky had a fluffy white cloud, hence the Microsoft naming logic.
Anyway onward, Microsoft Azure is a big deal in the computing cloud world and full stack web developers need to know about it. Blockchain, which is big news everywhere, was one of the big news previews highlighted this summer at the Microsoft Build developers conference, as Paul Schnackenburg explained in an in-depth Virtualization & Cloud Review article, A Look at the New Azure Releases & Improvements Revealed at Build 2019. There's a preview of Azure Blockchain Service providing point-and-click creation of fully managed networks with management and governance built-in. "There's also an Azure Blockchain Development Kit," he writes, "which brings connectors for Flow and LogicApps, another move to make adoption of this complex technology easier for non-experts. And the plug-in for Visual Studio Code is going to simplify development."
What's New for ASP.NET Core
With .NET Core 3.0 becoming production-ready, Microsoft is focusing on new features for ASP.NET including top level ASP.NET Core templates in Visual Studio: "The ASP.NET Core templates now show up as top level templates in Visual Studio in the 'Create a new project' dialog," Microsoft's Daniel Roth explained in a July Visual Studio Magazine article. What's New for ASP.NET Core offers summaries of the new features.
Working with Razor and Blazor
If you want to start working on the frontend, Visual Studio Magazine offers a handy how-to guide with code samples. In THE PRACTICAL CLIENT: How to Dynamically Build the UI in Blazor Components by Peter Vogel, he explains: "You have two tools for generating your initial UI in a Blazor component: ASP.NET's Razor and Blazor's RenderFragment." He shows how to use both to integrate with your C# code and also offers a caveat about what you can't do.
Angular 8 Grows Ivy
If you are angling for a full stack web developer position, you need to know your Angular. As a Wikipedia article explains: "Angular is a TypeScript-based open-source web application framework led by the Angular Team at Google and by a community of individuals and corporations." The latest version, Angular 8, was released this past May but the article notes that Google has promised new upgrades twice a year. The future of Angular is Ivy, which is a "backwards compatible, completely new render engine based on the incremental DOM architecture. Ivy has been engineered with tree shaking in mind, which means that application bundles will only include the parts of Angular source code that is actually used by the application." Angular 8 has an opt-in preview of Ivy with these features:
Generated code that is easier to read and debug at runtime.
- Faster re-build time.
- Improved payload size.
- Improved template type checking.
- Backwards compatibility
As an open source project there are lots of websites for Angular info including https://opensource.google.com/projects/angular, and https://github.com/topics/angular.
Pop Goes the Typescript
Get on Track for Full Stack
This blog offers a peek at what you need to know to be a full stack web developer. If you want to dive deep, sign up for VSLive! Chicago.
Posted by Richard Seeley on 08/27/20190 comments
If you are working with earlier versions of plain old ASP.NET, but need to upgrade your skills for Microsoft Azure DevOps, you may be looking to get up to speed on its smarter brother, ASP.NET Core, which is growing in popularity with the developer community.
While C# is the "most loved" programming language, according to a recent survey by developer tooling specialist JetBrains. There is growing regard for .NET Core, according to a recent Visual Studio Magazine article. "… the survey indicates Microsoft's new open source, cross-platform 'Core' direction is gaining traction but still has a long way to go as it usurps the ageing, Windows-only .NET Framework, with .NET Core and ASP.NET Core leading the migration," the article noted.
ASP.NET Core is "a complete rewrite that unites the previously separate ASP.NET MVC and ASP.NET Web API into a single programming model," according to a Wikipedia article. "ASP.NET Core applications support side by side versioning in which different applications, running on the same machine, can target different versions of ASP.NET Core. This is not possible with previous versions of ASP.NET."
Microsoft sought to distance the new framework from the older versions of ASP.NET. The company didn't want it to be thought of as simply an update, so the working title, ASP.NET 5 was changed to ASP.NET Core 1.0 for its 2016 release to highlight its status as a brand new product.
Microsoft has recently published an Introduction to ASP .NET Core that covers what developers need to know about the framework for building cloud-based, Internet-connected applications. It touts the framework's advantages including:
- Build web apps and services, IoT apps, and mobile backends.
- Use your favorite development tools on Windows, macOS, and Linux.
- Deploy to the cloud or on-premises.
- Run on .NET Core or .NET Framework.
Why ASP .NET Core?
Microsoft is aiming at ASP.NET 4.x developers, said to number in the millions, touting ASP .NET Core's ability to integrates seamlessly with popular client-side frameworks and libraries, including Blazor, Angular, React, and Bootstrap. The company also points out that the new framework provides benefits including:
Beyond explaining what the framework can do, the Microsoft tutorial page offers developers a step-by-step "learning path" with code samples for a basic app to show how it works.
Would you rather see how it's done rather than reading the instructions? There is a fun and informative one-hour YouTube video featuring Daniel Roth, program manager on Microsoft's ASP.NET team, covering "Full stack web development with ASP.NET Core 3.0 and Blazor." The highly entertaining host shows an enthusiastic live audience how to build a pizza store web app.
If videos are your thing, Microsoft's ASP .NET Community Standup site offers scheduled tutorials that you can watch live. All the past live events, several of them featuring the irrepressible Roth, are available for replay. Recent episodes also included the June ASP.NET Core 3.0 Preview 6 Release Party.
Hands-on Training for Building a Modern DevOps Pipeline
If you are looking for in-person hands-on training, ASP .NET Core will be in the spotlight on Sept. 29, at VS Live! San Diego. There will be a full-day hands-on lab: Building a Modern DevOps Pipeline on Microsoft Azure with ASP.NET Core and Azure DevOps.
Attendees will come away with an ASP.NET Core app and a SQL Server Database running in Azure with a full continuous integration / continuous deployment (CI/CD) pipeline managed by Azure DevOps.
The instructors will begin with a review of the current thinking on DevOps. Next will be the planning and tracking phase where the architecture of the app will be broken out and defined. Then the dev & test phase where attendees get feature flags implemented, CI builds working, manual and automated tests, and more. In the release phase, you will learn how to create a deployment pipeline to multiple environments and how to validate a deployment after its release using Azure App Services (both web apps and containers). Finally, the monitor and learn phase will cover analytics and user feedback and how you start the cycle over again.
By the end of the day, you will have a CI/CD pipeline configured, a deployed app, and the hands-on experience on how to build a modern ASP.NET Core and SQL Database solution that runs in Azure using Azure DevOps.
You can find out more and register here.
Posted by Richard Seeley on 07/19/20190 comments
At one time data was mostly a topic for database administrators. Not anymore. With artificial intelligence exploding on the scene, Big Data is a hot topic in the developer world.
The meteoric rise of the Internet of Things (IoT) is creating huge datasets – think terabytes and petabytes – that go beyond what traditional relational database management systems (RDBMS) and legacy software analytics tools can handle, according to a Wikipedia article. Sensory data is pouring in from IoT devices in industries including medical, manufacturing and transportation. That data is useful and sometimes even crucial but those industries need a way to make sense out of it.
A new generation of data analytics applications is needed to deal with what one analyst called the coming "datapocalypse." This presents a challenge and an opportunity for developers if they have the skills and tools to create those apps for business users.
On the tool front, .NET developers recently got good news with the preview of .NET for Apache Spark, which will allow them to more easily use the popular Big Data processing framework in C# and F# projects, according to an article by David Ramel, editor of Visual Studio Magazine.
"Spark is described as a unified analytics engine for large-scale data processing, compatible with Apache Hadoop data whether batched or streamed," the editor explained. "Currently, Spark is accessible via an interop layer with APIs for the Java, Python, Scala and R programming languages. While .NET coders have been able to use Spark with Mobius C# and F# language binding and extensions, the new project seeks to improve on that scheme while paving the way to add more language support."
In its announcement of .NET for Apache Spark, Microsoft said it "… provides high performance APIs for using Spark from C# and F#. With [these] .NET APIs, you can access all aspects of Apache Spark including Spark SQL, DataFrames, Streaming, MLLib etc. .NET for Apache Spark lets you reuse all the knowledge, skills, code, and libraries you already have as a .NET developer."
Microsoft’s .NET for Apache Spark website explains use cases:
- Large streams of data can be processed in real-time with Apache Spark, such as monitoring streams of sensor data or analyzing financial transactions to detect fraud.
- Apache Spark can reduce the cost and time involved in building machine learning models through distributed processing of data preparation and model training, in the same program.
- Modern business often requires analyzing large amounts of data in an exploratory manner. Apache Spark is well suited to the ad hoc nature of the required data processing.
As an open source project, Microsoft says that .NET, which is free, and now includes .NET for Apache Spark, requires no fees or licensing costs even for commercial projects.
There is a GitHub site with a tutorial for developers looking to get started with .NET for Apache Spark.
F# for Machine Learning
Speaking of F#, developers working with the open source, cross-platform language are getting new functionality for ML. The 15-year-old language currently works with Microsoft's ML.NET machine learning framework, but Microsoft says new ML functionality is in the works.
With the latest F# 4.6, Microsoft’s primary focus is on boosting performance for medium-to-large sized solutions, according to an article in Visual Studio Magazine.
"Other work included significant reductions in cache sizes, significant reductions in allocations when processing format strings, removing ambient processing of identifiers for suggestions when encountering a compile error, removing LOH allocations for F# symbols when they are finished being type-checked, and removing some unnecessary boxing of value types that are used in lots of IDE features," Microsoft said.
Updates to F# will now be synched with Visual Studio releases, according to a Microsoft announcement that concluded by telling developers: "With this in mind, you can think of the Visual Studio 2019 release and future updates as a continuous evolution of F# tooling."
ML for the Masses with Azure Update
Developers aren’t going to have all the fun in the AI revolution. The Azure Machine Learning Web UI is being updated for business power users who do not have programming skills, according to Microsoft.
"Emphasizing our mission to scale machine learning to the masses, we now introduce automated machine learning user interface (UI), which enables business domain experts to train ML models without requiring expertise in coding," said Tzvi Keisar, senior program manager, Microsoft Azure.
Find out more in this Visual Studio Magazine article.
Microsoft's 3 AI Dev Approaches
The Azure Machine Learning UI, is part of Microsoft’s three pronged approach to AI development, summed up as “Code First, No Code and Drag-and-Drop,” according to a recent Visual Studio Magazine article.
This three-prong approach was outlined by Bharat Sandhu, director of artificial intelligence at Microsoft, to fit different classifications of developers, or "AI authoring models:"
- Code first: use any tools
- No code: use automated machine learning
- Drag and drop: make models visually
Explaining the three approaches for different types of AI authors, Microsoft sees:
- Developers and data scientists who want to write code to build machine learning models. They will take the code first model Azure Machine Learning offers.
- Business domain experts, who know data, but don't know much about machine learning or code will use Azure Machine Learning's automated machine learning 'no code' option.
- IT professionals and experts in statistics or mathematics, who are not coders but want to make their own models, will use a drag-and-drop approach.
So Microsoft is planning a big tent approach to AI development to accommodate as many developers and power users as possible.
AI, Data and ML at VSLive! at Microsoft Headquarters
If you are a developer, who wants to up your skills, AI, Big Data and Machine Learning will be a hot topic at VS Live! Microsoft HQ in Redmond WA, Aug. 12 – 16. Sessions will cover:
- AI and analytics with Apache Spark and Azure Databricks
- Deep learning for developers
- Data pipelines and analytics on Azure
- SQL Server 2019 deep dive
- Azure Cosmos DB
- Power BI
Find out more and sign up here.
Posted by Richard Seeley on 06/19/20190 comments
Everybody involved with any kind of software development has heard the DevOps mantra: "DevOps automates and speeds software delivery." Of course, not everyone practices DevOps because there’s a learning curve and a discipline to follow and that’s not always easy especially in organizations demanding instant application gratification. Despite the hype, getting started in DevOps can be a challenge.
So beyond marketing mantras, what positive things can DevOps do for you?
"DevOps is a practice that unifies people, process, and technology across development and IT in five core practices: planning and tracking, development, build and test, delivery, and monitoring and operations," Microsoft explains in an overview of Azure DevOps. "When practicing DevOps, development, IT operations, quality engineering, and security teams work closely together—breaking down practices that were once siloed. Improved coordination and collaboration across these disciplines reduces the time between when a change is committed to a system and when the change is placed into production. And, it ensures that standards for security and reliability are met as part of the process. The result: better products, delivered faster ..."
If you get the time and training to implement it, DevOps could help satisfy demands for instant application gratification.
There are lots of tools to help developers but for someone new to DevOps, it’s not always clear what does what.
In the Microsoft cloud world, Azure DevOps server and services is designed to smooth the developer’s way into the paradigm.
"Azure offers an end-to-end, automated solution for DevOps that includes integrated security and monitoring," according to Microsoft. "The developer experience of Azure DevOps integrates with the tools of your choice. If you’re a Java developer—great—Azure provides native integrations with Eclipse. If you build with Jenkins, use it to easily deploy directly to Azure. Bring your development, IT operations, and quality engineering teams together to build, test, deploy, monitor and manage applications in the cloud."
Deciphering Microsoft Names
The products sound good but the product naming can get a little confusing. Things have evolved. First of all there is Azure DevOps Services, a subscription-based Azure platform, announced in September 2018, that's hosted from Microsoft's datacenters. Then this March, the Redmond, WA-based company announced Azure DevOps Server 2019, which is designed for organizations that want to deploy it in their own infrastructures or any datacenter. One more clarification is needed because Azure DevOps Server 2019 is replacing Team Foundation Server (TFS). Microsoft says that "Azure DevOps represents the evolution of Visual Studio Team Services (VSTS)" and suggests users of Team Foundation Server 2012 and newer versions upgrade to Azure DevOps Server 2019.
Now, that that’s cleared up, what do these versions of the platform provide for developers wanting to apply the DevOps model to their Azure-based products?
In a blog Introducing Azure DevOps Service, Jamie Cool, Director of Program Management, Azure DevOps, said: "Working with our customers and developers around the world it’s clear DevOps has become increasingly critical to a team’s success. Azure DevOps captures over 15 years of investment and learnings in providing tools to support software development teams."
In announcing the Azure DevOps Server, Cool said it "includes developer collaboration tools which can be used together or independently, including Azure Boards (Work), Azure Repos (Code), Azure Pipelines (Build and Release), Azure Test Plans (Test), and Azure Artifacts (Packages). These tools support all popular programming languages, any platform (including macOS, Linux, and Windows) or cloud, as well as on-premises environments."
As with TFS, developers control where they install Azure DevOps Server and when they apply updates.
Cool says there are major updates from TFS 2018 to Azure DevOps Server 2019, and his key highlights include:
- The new navigation, which enables users to easily navigate between services, is more responsive and provides more space to focus on your work.
- Azure Pipelines has been enhanced in many ways including new Build and Release pages, and support for YAML builds.
And for those who would rather let Microsoft manage the maintenance and updates, Cool points out that Azure DevOps Services is available.
"Each Azure DevOps service is open and extensible," according to Microsoft. "You can use them together for a full DevOps solution or with other services. If you want to use Azure Pipelines to build and test a Node service from a repo in GitHub and deploy it to a container in AWS, go for it. Azure DevOps supports both public and private cloud configurations. Run them in our cloud or in your own data center."
Azure and AI
As part of its big Build developer conference earlier this month, Microsoft announced new development features for Azure related to artificial intelligence development with machine learning.
As reported in a Visual Studio Magazine article, the news includes MLOps (DevOps for machine learning) capabilities with Azure DevOps integration. MLOps is designed to provide developers with reproducibility, auditability and automation of the end-to-end machine learning lifecycle.
"We're delivering key new innovations in Azure Machine Learning that simplify the process of building, training and deployment of machine learning models at scale," exec Scott Guthrie announced in a blog post before the conference. "These include new automated machine learning advancements and an intuitive UI that make developing high-quality models easier, a new visual machine learning interface that provides a zero-code model creation and deployment experience using drag-and-drop capabilities and new machine learning notebooks for a rich, code-first development experience."
DevOps in the Spotlight
The spotlight will be on DevOps at VSLive! in Boston, June 9 – 13. It will include a Full Day Hands-On Lab: Building a Modern DevOps Pipeline on Microsoft Azure with ASP.NET Core and Azure DevOps.
"By the end of the day you'll have your own Azure DevOps organization with a CI/CD pipeline configured, a deployed app, and the hands-on experience on how to build a modern ASP.NET Core and SQL Database solution that runs in Azure using Azure DevOps," the lab description promises.
Find out about all the learning opportunities at vslive! in boston.
Posted by Richard Seeley on 05/15/20190 comments
DevOps can be a challenge especially for developers starting out with it. But a new Basic Process in Azure DevOps aims to pare down the more complicated methodologies.
On an April 2019 Microsoft video explaining how Basic Process works, Dan Hellem, program manager on the Azure DevOps team, told viewers that the goal is to not only attract engineers to the product but also keep them as loyal users: "We’ve been talking about putting out the Basic Process for several years. Looking at it from the perspective of a new person coming into Azure DevOps, we have three processes: Agile, Scrum and CMMI [Capability Maturity Model Integration]. The problem with those processes is they are very methodology heavy."
In a February 2019 blog introducing Basic Process, Hellem noted that Agile is the most popular of the three processes but even it can be difficult for newcomers.
"The Agile process still brings a set of concepts and behaviors that are not obvious to our new users and therefore some of those users have a hard time understanding Azure Boards (Basic Process)," he wrote. "For example, the four-level backlog hierarchy or the many state transition rules. These add complexities that new users don’t care about. New users come from tools such as GitHub with very simple work tracking, and they want Azure Boards to be just as easy."
In the video, he imagines how Azure DevOps may seem to the first-time user: "So if I’m a new engineer and I’m working with GitHub stuff and I come over and want to use Azure DevOps, are you really going to make me work with product backlog items and user stories and all those weird those weird terminologies? What we found is that when engineers start using the product, they start dropping off because it is just too complicated. We want engineers to use the product, so what we did is get rid of the baggage like Agile and Scrum and CMMI. We wanted to make it easier to work with so engineers can start working with it and start getting things done."
Keeping it Simple
To achieve their goal the Azure DevOps team reduced the scope of the process to get down to the basics, Hellem explained in his blog. "To start we reduced the number of work item types down to three: Epic, Issue, and Task. By default, users can start right away by adding Issues to their board. You will notice that the board contains 3 columns. To Do, Doing, and Done. This simplified state model is used for all three work item types."
Work item types in Agile, Scrum and CMMI contain many extra fields that are not needed for someone starting out, Hellem explained. "Our research shows that many users get confused by all of the extra fields and their purpose. In the Basic Process, we kept only the core fields and removed the rest. Only fields that are required to support other functionality survived."
Hierarchy is the last area the Azure DevOps team simplified in the Basic Process. "Instead of four levels, Basic starts with just two," Hellem explained. "Users will start with Issues, and those Issues can be broken down into Tasks. For more advanced scenarios, issues may not be enough. Some users may want a way to group their issues into specific deliverables. For these users we are providing the Epic work item type."
Beyond his blog, Hellem encourages developers to read the documentation published in January 2019, Start using Azure Boards (Basic Process). Talk about simple, the documentation only takes about two minutes to read.
Other Azure DevOps Improvements
It’s been a busy few months for Azure DevOps.
In March, Microsoft announced the commercial release of the new Azure DevOps Server 2019.
"DevOps Server 2019, used for developer collaboration, is the company's rebranded successor to Team Foundation Server 2018," explained Kurt Mackie, in an Application Development Trends (ADT.mag) article on the announcement.
"Azure DevOps Server 2019 is notable for having a redesigned user interface that follows Microsoft's Fluent design concepts," Mackie explained. "Developers get access to various services, such as Azure Pipelines for continuous integration/continuous development across different languages and platforms. It has an Azure Artifacts service for package feeds and project tracking via the Azure Boards service. Testing is supported by the Azure Test Plans component. The server also works with the Azure Repos service to integrate with Git repos."
DevOps Specialization Pays
If you think DevOps isn’t that interesting, think again. For DevOps specialists, job satisfaction is high and so is the pay.
A survey of 88,000 developers around the world, released April 9 by Stack Overflow, found that people who know DevOps are well paid and happy in their work.
In an article on the survey, David Ramel, editor of Visual Studio magazine, wrote: "Culling through that data finds one main takeaway about the DevOps movement that has been gaining steam by providing automated processes that bridge software development and IT teams to improve the build/test/release cycle:
DevOps specialists and site reliability engineers are among the highest paid, most experienced developers most satisfied with their jobs …
If you want to check out all the DevOps survey results, it’s available here.
DevOps in the Spotlight
Want to get on the DevOps bandwagon? One of the highlights at the upcoming VSLive! in Boston, happening June 9 through 13, will be an special track on DevOps.
See what the track description offers.
DevOps in the Spotlight: You have a role to play when it comes to DevOps, and in this track, you'll learn the about the tools, techniques, and concepts that you can immediately apply to your daily work.
You'll find coverage of:
- Azure DevOps Services in the cloud
- Azure DevOps Server on premises (formally named Team Foundation Server)
- Writing maintainable test automation
- Architecting solutions for DevOps and continuous delivery
- Getting started with git
- Azure Secure DevOps
- Database DevOps
Find out more here.
Posted by Richard Seeley on 04/17/20190 comments
Service-oriented architecture (SOA) and web services were hot developer trends a decade ago but may only get a yawn today. Now, microservices take the concept into the cloud and container era.
If you missed the SOA and web services hoopla, and haven’t worked with microservices here’s a brief explanation from Wikipedia:
“Microservices are a software development technique—a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services.”
Microservices and containers are growing in popularity because the modular approach is better suited to today’s dynamic business needs than older programming methodologies with zillions of lines of code.
As Michael Otey explained in an in-depth Redmond Magazine article: “Traditional monolithic applications consist of large executable programs that are complex and can be difficult to deploy and update. In contrast, container-based microservice applications are composed of many small independent services running in containers that enable the organization to deploy and update granular microservices individually without impacting the entire application. This microservice architecture potentially makes these applications more resilient, as well as easier to deploy and update.”
If you want to take a deeper dive into microservices as well as containers, there are resources available to help you get up to speed.
Visual Studio users working with .NET can download a free 340-page eBook from Microsoft, .NET Microservices: Architecture for Containerized .NET Applications. Reading and digesting 340 pages of material is going to be a challenge. The authors offer some Key Takeaways to provide Microsoft’s overview of this way of doing application development.
Making the case for microservices architecture, the authors assert that it is “becoming the preferred approach for distributed and large or complex mission-critical applications based on many independent subsystems in the form of autonomous services. In a microservice-based architecture, the application is built as a collection of services that are developed, tested, versioned, deployed, and scaled independently. Each service can include any related autonomous database.”
For developers who have been working with SOA, this Lego approach to building an application from existing pieces of software may be old hat. One new wrinkle since SOA emerged in 1998, is containers, specifically the Docker technology that was first introduced in 2013. For cloud applications, microservices and containers go together, although containers also have wider utility in the .NET world, as the authors of the Microsoft eBook explain: “Containers are convenient for microservices, but can also be useful for monolithic applications based on the traditional .NET Framework, when using Windows Containers. The benefits of using Docker, such as solving many deployment-to-production issues and providing state-of-the-art Dev and Test environments, apply to many different types of applications.”
As one of the major contributors to the open source container project, Microsoft is pretty much all in with Docker when it comes to microservices architecture. “Docker-based containers are becoming the de facto standard in the industry, supported by key vendors in the Windows and Linux ecosystems, such as Microsoft, Amazon AWS, Google, and IBM. Docker will probably soon be ubiquitous in both the cloud and on-premises datacenters,” the eBook authors write.
But you don’t have to take Microsoft’s word for it. On its website, Docker is touting Forrester Research Inc.’s latest take on the technology, New Wave Enterprise Container Platform, Q4 2018 Report. Docker “leads the pack with a robust container platform well-suited for the enterprise,” the container company says.
“The purpose of Docker is to build containers that hold, potentially, all of the components of an application: the application itself, the database engine, any Web services it requires and so on. That container, unlike a virtual machine, doesn't require an operating system so it takes less space than a VM and starts up/shuts down faster,” explained Peter Vogel, in a recent Visual Studio Magazine article, Understanding Docker Vocabulary.
For those starting out with using containers for their applications, Vogel said there’s good news. “There are a bunch of prepared containers waiting for you to use on Docker Hub: these are called images. Many of them are Linux based, but for .NET Core applications that's not an issue: Core runs as well on Linux as Windows.”
While Docker is given credit for popularizing containers, Vogel notes that when it comes to container orchestration the “elephant in this living room is Kubernetes.”
Kubernetes, first released in 2015, is an open source container orchestration system developed by Google but also championed by Microsoft.
In his Redmond article on containers referenced above, Otey explained: “Kubernetes provides automated deployment, scaling and operations for containers. It provides a management control plane for containers that works above the container level … Essentially, Kubernetes is used for managing distributed application containers across clusters of physical or virtual machines. It supports a range of different container tools including support for Docker. Kubernetes allows you to combine multiple containers that make up an application into logical groups for easier management. It's designed to enable you to easily deploy, update and scale your applications, as well as optionally limiting hardware utilization.”
The article explained that Kubernetes is the Greek word for helmsman or pilot. For non-Greek speakers who have trouble pronouncing the name, Otey said “Kubernetes is sometimes referred to as K8S, where the eight letters of ubernete are replaced by the number 8.
Visual Studio Live! New Orleans: Cloud, Containers and Microservices
Reading documentation on application development approaches is good but it is hard to beat in-person learning from experts who really know their stuff. Visual Studio Live! in New Orleans, April 22 to 26, is offering a track covering Cloud, Containers and Microservices including:
- Microservice architectures
- Containers 101
- Azure Kubernetes Services
- Azure Active Directory B2C
- Serverless and Azure Functions
- Intelligent apps in the cloud with Azure Cognitive Services
Find out more.
Posted by Richard Seeley on 03/20/20190 comments
"I will talk about two sets of things. One is how productivity and collaboration are reinventing the nature of work, and how this will be very important for the global economy. And two, data. In other words, the profound impact of digital technology that stems from data and the data feedback loop." ~ Microsoft CEO Satya Nadella
Collaboration, productivity and data are what Azure Databricks is all about.
Data, of course, is everything. How many F150 trucks did Ford build this year? What is the patient’s heart rate and blood pressure? Where can we find sushi at this time of night?
The iPhone in your back pocket is filled with data and searching for more.
How does that data get organized so you can find it when you need it?
Machines do a lot of that work. But people working in collaboration with other people and other machines make the data driven world go around.
Azure Databricks is a collaboration between Microsoft and the creators of Apache Spark, which is described on its homepage as an "analytics engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing."
Databricks, the company created to commercialize Spark, "provides a Unified Analytics Platform for data science teams to collaborate with data engineering and lines of business to build data products," the company states. "Users achieve faster time-to-value with Databricks by creating analytic workflows that go from ETL and interactive exploration to production. The company also makes it easier for its users to focus on their data by providing a fully managed, scalable, and secure cloud infrastructure that reduces operational complexity and total cost of ownership."
One Click Setup and Management
In announcing the collaboration with Databricks in 2017, Microsoft touted Azure Databricks as "a fast, easy and collaborative Apache Spark-based analytics platform that delivers one-click setup, streamlined workflows and an interactive workspace. Native integration with Azure SQL Data Warehouse, Azure Storage, Azure Cosmos DB, Azure Active Directory and Power BI simplifies the creation of modern data warehouses that enable organizations to provide self-service analytics and machine learning over all data with enterprise-grade performance and governance."
In a Microsoft overview of Azure Databricks, the company explains its value-add: "Azure Databricks features optimized connectors to Azure storage platforms (e.g. Data Lake and Blob Storage) for the fastest possible data access, and one-click management directly from the Azure console. This is the first time that an Apache Spark platform provider has partnered closely with a cloud provider to optimize data analytics workloads from the ground up."
What Does All This Mean?
So here’s this extensive set of data tools, what can you build with them? In January, Databricks provided answers from data industry thought leaders, who focused on the need for solutions to issues organizations face with AI, Big Data and Analytics.
Kamelia Aryafar, chief algorithm officer at Overstock, sees deep learning, which is a class of machine learning algorithms facilitated by the Spark technology, paying dividends for organizations. "Deep learning innovations will create a lot of new AI applications, some of which are already in production and making massive changes in the industry," she is quoted as saying. She noted that Overstock is currently using deep learning to improve marketing projects such as email campaigns.
Other thought leaders quoted by Databricks see the need for the latest data tools to be used to improve long-standing issues including data processing and providing trusted data with "Explainable AI."
Because of the social, economic and commercial implications of the data being generated, "it is critical to develop AI that is explainable, provable and transparent," said Mainak Mazumdar, chief research officer at Nielsen.
Databricks CEO and co-founder Ali Ghodsi finds data processing to still be a challenge in the AI era. "As an industry we tend to believe that data scientists are spending the majority of their time developing models, shares. Truth be told, data processing remains the hardest and most time consuming part of any AI initiative. The highly iterative nature of AI forces data teams to switch between data processing tools and machine learning tools. For organizations to succeed at AI in 2019, they have to leverage a platform that unifies these disparate tools."
Reading between the lines, Databricks provides the platform companies need to leverage.
Databricks concluded the survey on the near future of AI and Big Data, stating: “Solving the world’s toughest data problems starts with bringing all of the data teams together within an organization. Data science and engineering teams’ ability to innovate faster has historically been hindered by poor data quality, complex machine learning tool environments, and limited talent pools. Additionally, organizational separation creates friction and slows projects down, becoming an impediment to the highly iterative nature of AI projects. Much like in 2018, organizations that leverage Unified Analytics will have a competitive advantage with the ability to build data pipelines across various siloed data storage systems and to prepare labelled datasets for model building, which allows organizations to do AI on their existing data and iteratively do AI on massive data sets.”
Practical Use Cases
Healthcare is an area where AI can be used to parse patient data to provide diagnostic and other assistance to medical professionals.
Last June, Databricks announced that it has been working with pharmaceutical and healthcare providers "to improve their drug discovery processes."
"One such customer, the Regeneron Genetics Center (a wholly-owned subsidiary of Regeneron, a leading biotechnology company), has sequenced over 300,000 consented volunteers and paired their de-identified genetic data with de-identified electronic health records to uncover actionable insights for drug discovery and development,” Databricks said in the announcement.
Jeffrey Reid, PhD, Head of Genome Informatics at Regeneron, was quoted as saying: "As this dataset has grown rapidly, we encountered significant barriers in simple tasks, like gathering all of the data for a given analysis, and querying the 10s of billions of results from our studies. Not only has the Databricks Unified Analytics Platform solved these big data problems, but it is enabling everyone in our integrated drug development process – from physician-scientists to computational biologists – to easily access, analyze, and extract insights from all of our data. Drug development is still a long and difficult process rife with failure, but we have already significantly reduced the amount of time it takes to generate important early insights."
Databricks cited the following areas where its platform enables medical researchers to:
- Accelerate discovery with simplified genomic pipelines: Simplify workflows with prebuilt genomic pipelines hosted in the cloud to process large datasets up to 100x faster than existing solutions.
- Innovate faster with interactive, tertiary analytics and AI at scale: Quickly and simply run tertiary analytics and machine learning algorithms on massive genomic datasets with prepackaged frameworks designed to run in parallel.
- Improve productivity across data, analytics and research teams: Create a collaborative environment and shared workspaces for bioinformaticians, computational biologists and researchers to work together across the research lifecycle with shared workspaces, saving teams precious time and resources.
Training for AI, Data, and Machine Learning
Working with AI, Big Data and Machine Learning is the future of application development. If you want to build Azure Databricks skills, Visual Studio Live! New Orleans this April offers a session on AI and Analytics with Apache Spark on Azure Databricks where you will learn:
- About the fundamentals of Apache Spark, Spark SQL and Spark MLlib
- How to use Databricks notebooks
- How to manage clusters and jobs
- How to integrate Azure Databricks with blob storage and Azure Data Lake Store (ADLS)
- How to write Python code for both analytics and machine learning
Find out more here.
Posted by Richard Seeley on 02/20/20190 comments