UI components bundle

Modular Design

Getting Started

Pytheas project contains a simple helloworld application that serves as a template for building new applications using the framework. Please refer to instructions on how to run helloworld application from a command line.

In short, “garbage collection visualization” (hereafter shortened to “gcviz”) is the process of turning gc.log[1] into x/y time-series scatter plots (pictures). That is, turning garbage collector (GC) logging output into two types of charts, GC event charts:

and heap size charts:

Setting the context for gcviz

• hand-crafting excel spreadsheets
• using PrintGCStats/PrintGCFixup/GCHisto
• visually skimming gc.log, looking for “long” stop-the-world events

• Convenient: The developer loads the event and heap chart pages directly into their browser by clicking on a link in Asgard. gcviz does all the processing behind the scenes. This convenience allows us to quickly reinforce or reject the oft-repeated claim: bad application behavior is "because of long GC pauses".

• Visual: Images can convey time-series information more quickly and densely than text.
• Causation: gcviz can help application developers establish or refute a correlation between time-of-day-based observed application degradation and JVM-wide GC behavior.
• Clarity: The semantics of each of the different GC event types (concurrent-mark, ParNew) can be made implicit in the image instead of being given equal weight in the text of gc.log. This is useful because, for example, a long running concurrent-sweep should be no cause for alarm because it runs in parallel with the application program.
• Iterative: gcviz allows for quick interpretation of experimental results. What effect does modifying a particular setting or algorithm have on the heap usage or GC subsystem? gcviz allows one to quickly and visually understand any impact a GC-related change has made. gcviz is now being effectively used in canary deployments within Netflix where GC behavior is compared to an existing baseline.

• AppDynamics http://www.appdynamics.com/
• GarbageCat http://code.google.com/a/eclipselabs.org/p/garbagecat/
• GCHistohttp://java.net/projects/gchistohttps://gchisto.dev.java.net/
• gcviewerhttps://github.com/chewiebug/GCViewer
• GCViewer http://www.tagtraum.com/gcviewer.html
• HP jmeterhttps://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER
• IBM GCMV http://www.ibm.com/developerworks/java/jdk/tools/gcmv/
• IBM pmathttps://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=22d56091-3a7b-4497-b36e-634b51838e11
• jClarity Censum http://www.jclarity.com/products/censum/
• jvisualvm http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/index.html
• PrintGCFixup http://article.gmane.org/gmane.comp.java.openjdk.hotspot.gc.devel/51/match=gc+log+reader
• PrintGCStats http://java.net/projects/printgcstats/
• Shrek: http://www.slideshare.net/arunkejariwal/a-tool-for-practical-garbage-collection-analysis-in-the-cloud
• verbosegcanalyzer http://code.google.com/p/verbosegcanalyzer/

• Run outside the context of the application under analysis, but on each instance on which the application runs.
• Use gc.log as the source-of-truth
• historical analysis (no need to attach to a running vm)
• filesystem/logs can be read even when HTTP port is busy and/or application threads are blocked
• Read gc logs with the JVM options that Netflix commonly (but not universally) uses:
• -verbose:gc
• -verbose:sizes
• -Xloggc:/apps/tomcat/logs/gc.log
• -XX:+PrintGCDetails
• -XX:+PrintGCDateStamps
• -XX:+PrintTenuringDistribution
• -XX:-UseGCOverheadLimit
• -XX:+ExplicitGCInvokesConcurrent
• -XX:+UseConcMarkSweepGC
• -XX:+CMSScavengeBeforeRemark
• -XX:+CMSParallelRemarkEnabled

• Gracefully handle any arguments passed to the JVM.
• Remain independent of any special terminal/display requirements (must be able to run inside and outside Netflix’s base AMI without modification)

• Be able to run standalone to leverage special display capabilities if they are present
• Be accessible from Asgard
• Be dead-simple to use, requiring no specialized gc knowledge
• Correlate netflix-internal events with gc activity. One example of a netflix-internal event is a cache refresh.
• Retain its reports over time to enable long-term comparative analysis of a given application.

A Small Example: GC Event Chart

• a DefNew GC event at 60 seconds after JVM boot,
• a DefNew GC event at 120 seconds after JVM boot, and
• a DefNew GC event at 180 seconds after JVM boot

Real-world GC Event Charts

In mid-2012, Netflix used gcviz to analyze some performance problems a class of applications was having. These performance problems had 80-120 second garbage collection pauses as one of its symptoms. The gcviz event chart looked like this:

After the problem was identified and fixed the event chart looked like this:

In another mid-2012 event, the heap usage of a Netflix application went higher than the designers of that application intended. The application designers intended memory usage to peak around 15GB. The chart below shows that heap usage peaked at 2.0 times 10^7 kilobytes (19.07 gigabytes) and it also shows “flumes” for periods of time where the heap usage went above its previous/standard high-water mark:

after that problem was identified and fixed the high-water mark returned to 1.6 times 10^7 kilobytes (15.23 gigabytes) and the flumes reduced and the heap usage pattern became far more regular. In addition the low-water mark dropped from 11.44 gigabytes to 9.54 gigabytes:

JVM collector compatibility

Additionally collected data

Open-sourcing details

Zuul in Netflix's Cloud Architecture

How Does Zuul Work?

• Type: most often defines the stage during the routing flow when the filter will be applied (although it can be any custom string)
• Execution Order: applied within the Type, defines the order of execution across multiple filters
• Criteria: the conditions required in order for the filter to be executed
• Action: the action to be executed if the Criteria are met

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

classDeviceDelayFilterextends ZuulFilter {

defstatic Random rand =new Random()

@Override
     String filterType(){
return'pre'
}

@Override
intfilterOrder(){
return5
}

@Override
booleanshouldFilter(){
return  RequestContext.getRequest().
     getParameter("deviceType")?equals("BrokenDevice"):false
}

@Override
    Object run(){
 sleep(rand.nextInt(20000))//Sleep for a random number of seconds
//between [0-20]
}
}

Zuul Core Architecture

• PRE filters execute before routing to the origin. Examples include request authentication, choosing origin servers, and logging debug info.
• ROUTING filters handle routing the request to an origin. This is where the origin HTTP request is built and sent using Apache HttpClient or Netflix Ribbon.
• POST filters execute after the request has been routed to the origin.  Examples include adding standard HTTP headers to the response, gathering statistics and metrics, and streaming the response from the origin to the client.
• ERROR filters execute when an error occurs during one of the other phases.

Request Lifecycle

How We Use Zuul

• Authentication
• Insights
• Stress Testing
• Canary Testing
• Dynamic Routing
• Load Shedding
• Security
• Static Response handling
• Multi-Region Resiliency 

Insights

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

classSharpDebugFilterextends SurgicalDebugFilter {
privatestaticfinal Set<String> DEVICE_IDS =["XXX","YYY","ZZZ"]
@Override
booleanpatternMatches(){
final RequestContext ctx = RequestContext.getCurrentContext()
final String requestUri = ctx.getRequest().getRequestURI();
final String contextUri = ctx.requestURI;
       String id = HTTPRequestUtils.getInstance().
           getValueFromRequestElements("deviceId");
return DEVICE_IDS.contains(id);
}
}

Stress Testing 

Multi-Region Resiliency

Zuul OSS

• zuul-core - A library containing a set of core features.
• zuul-netflix - An extension library using many Netflix OSS components:
• Servo for insights, metrics, monitoring
• Hystrix for real time metrics with Turbine
• Eureka for instance discovery
• Ribbon for routing
• Archaius for real-time configuration
• Astyanax for and filter persistence in Cassandra
• zuul-filters - Filters to work with zuul-core and zuul-netflix libraries 
• zuul-webapp-simple -  A simple example of a web application built on zuul-core including a few basic filters
• zuul-netflix-webapp- A web application putting zuul-core, zuul-netflix, and zuul-filters together.

Netflix OSS libraries in Zuul

• Weighted load balancing to balance a percentage of load to a certain server or cluster for capacity testing
• Request debugging
• Routing filters for Apache HttpClient and Netflix Ribbon
• Statistics collecting

• Accounts
• Regions
• Services (e.g. EC2, S3, EBS)
• Usage types (e.g. EC2 - m1.xlarge)
• Cost and Usage Categories (On-Demand, Un-Used, Reserved, etc.)

by Sriram Krishnan

In a prior tech blog, we had discussed the architecture of our petabyte-scale data warehouse in the cloud. Salient features of our architecture include the use of Amazon’s Simple Storage Service (S3) as our "source of truth", leveraging the elasticity of the cloud to run multiple dynamically resizable Hadoop clusters to support various workloads, and our horizontally scalable Hadoop Platform as a Service called Genie.

Today, we are pleased to announce that Genie is now open source, and available to the public from the Netflix OSS GitHub site.

What is Genie?

Genie provides job and resource management for the Hadoop ecosystem in the cloud. From the perspective of the end-user, Genie abstracts away the physical details of various (potentially transient) Hadoop resources in the cloud, and provides a REST-ful Execution Service to submit and monitor Hadoop, Hive and Pig jobs without having to install any Hadoop clients. And from the perspective of a Hadoop administrator, Genie provides a set of Configuration Services, which serve as a registry for clusters, and their associated Hive and Pig configurations.

Why did we build Genie?

There are two main reasons why we built Genie. Firstly, we run multiple Hadoop clusters in the cloud to support different workloads at Netflix. Some of them are launched as needed, and are hence transient - for instance, we spin up “bonus” Hadoop clusters nightly to augment our resources for ETL (extract, transform, load) processing. Others are longer running (viz. our regular “SLA” and “ad-hoc” clusters) - but may still be re-spun from time to time, since we work under the operating assumption that cloud resources may go down at any time. Users need to discover the latest incarnations of these clusters by name, or by the type of workloads that they support. In the data center, this is generally not an issue since Hadoop clusters don’t come up or go down frequently, but this is much more common in the cloud.

Secondly, end-users simply want to run their Hadoop, Hive or Pig jobs - very few of them are actually interested in launching their own clusters, or even installing all the client-side software and downloading all the configurations needed to run such jobs. This is generally true in both the data center and the cloud. A REST-ful API to run jobs opens up a wealth of opportunities, which we have exploited by building web UIs, workflow templates, and visualization tools that encapsulate all our common patterns of use.

What Genie Isn’t

Genie is not a workflow scheduler, such as Oozie. Genie’s unit of execution is a single Hadoop, Hive or Pig job. Genie doesn’t schedule or run workflows - in fact, we use an enterprise scheduler (UC4) at Netflix to run our ETL.

Genie is not a task scheduler, such as the Hadoop fair share or capacity schedulers either. We think of Genie as a resource match-maker, since it matches a job to an appropriate cluster based on the job parameters and cluster properties. If there are multiple clusters that are candidates to run a job, Genie will currently choose a cluster at random. It is possible to plug in a custom load balancer to choose a cluster more optimally - however, such a load balancer is currently not available.

Finally, Genie is not an end-to-end resource management tool - it doesn’t provision or launch clusters, and neither does it scale clusters up and down based on their utilization. However, Genie is a key complementary tool, serving as a repository of clusters, and an API for job management.

How Genie Works

The following diagram explains the core components of Genie, and its two classes of Hadoop users - administrators, and end-users.

Genie itself is built on top of the following Netflix OSS components:

• Karyon, which provides bootstrapping, runtime insights, diagnostics, and various cloud-ready hooks,
• Eureka, which provides service registration and discovery,
• Archaius, for dynamic property management in the cloud,
• Ribbon, which provides Eureka integration, and client-side load-balancing for REST-ful interprocess communication, and
• Servo, which enables exporting metrics, registering them with JMX (Java Management Extensions), and publishing them to external monitoring systems such as Amazon's CloudWatch.

Genie can be cloned from GitHub, built, and deployed into a container such as Tomcat. But it is not of much use unless someone (viz. an administrator) registers a Hadoop cluster with it. Registration of a cluster with Genie is as follows:

• Hadoop administrators first spin up a Hadoop cluster, e.g. using the EMR client API.
• They then upload the Hadoop and Hive configurations for this cluster (*-site.xml’s) to some location on S3.
• Next, the administrators use the Genie client to discover a Genie instance via Eureka, and make a REST-ful call to register a cluster configuration using a unique id, and a cluster name, along with a few other properties - e.g. that it supports “SLA” jobs, and the “prod” metastore. If they are creating a new metastore configuration, then they may also have to register a new Hive or Pig configuration with Genie.

After a cluster has been registered, Genie is now ready to grant any wish to its end-users - as long as it is to submit Hadoop jobs, Hive jobs, or Pig jobs!

End-users use the Genie client to launch and monitor Hadoop jobs. The client internally uses Eureka to discover a live Genie instance, and Ribbon to perform client-side load balancing, and to communicate REST-fully with the service. Users specify job parameters, which consist of:

• A job type, viz. Hadoop, Hive or Pig,
• Command-line arguments for the job,
• A set of file dependencies on S3 that can include scripts or UDFs (user defined functions).

Users must also tell Genie what kind of Hadoop cluster to pick. For this, they have a few choices - they can use a cluster name or a cluster ID to pin to a specific cluster, or they can use a schedule (e.g. SLA) and a metastore configuration (e.g. prod), which Genie will use to pick an appropriate cluster to run a job on.

Genie creates a new working directory for each job, stages all the dependencies (including Hadoop, Hive and Pig configurations for the chosen cluster), and then forks off a Hadoop client process from that working directory. It then returns a Genie job ID, which can be used by the clients to query for status, and also to get an output URI, which is browsable during and after job execution (see below). Users can monitor the standard output and error of the Hadoop clients, and also look at Hive and Pig client logs, if anything went wrong.

The Genie execution model is very simple - as mentioned earlier, Genie simply forks off a new process for each job from a new working directory. Other than simplicity, important benefits of this approach include isolation of jobs from each other and from Genie, and easy accessibility of standard output, error and job logs for our end-users (since they are browsable from the output URIs). We made a decision not to queue up jobs in Genie - if we had implemented a job queue, we would have had to implement a fair-share or capacity scheduler for Genie as well, which is already available at the Hadoop level. The downside of this approach is that a JVM is spawned for each job, which implies that Genie can only run a finite number of concurrent jobs on an instance, based on available memory.

Deployment at Netflix

Genie scales horizontally using ASGs (Auto-Scaling Groups) in the cloud, which helps us run several hundreds of concurrent Hadoop jobs in production at Netflix, with the help of Asgard for cloud management and deployment. We use Asgard (see screenshot below) to pick minimum, desired and maximum instances (for horizontal scalability) in multiple availability zones (for fault tolerance). For Genie server pushes, Asgard provides the concept of a “sequential ASG”, which lets us route traffic to new instances of Genie once a new ASG is launched, and turn off traffic to old instances by marking the old ASG out of service.

Using Asgard, we can also set up scaling policies to handle variable loads. The screenshot below shows a sample policy, which increases the number of Genie instances (by one) if the average number of running jobs per instance is greater than or equal to 25.

Usage at Netflix

Genie is being used in production at Netflix to run several thousands of Hadoop jobs daily, processing hundreds of terabytes of data. The screenshot below (from our internal Hadoop investigative tool, code named “Sherlock”) shows some of our clusters over a period of a few months.

The blue line shows one of our SLA clusters, while the orange line shows our main ad-hoc cluster. The red line shows another ad-hoc cluster, with a new experimental version of a fair-share scheduler. Genie was used to route jobs to one of the two ad-hoc clusters at random, and we measured the impact of the new scheduler on the second ad-hoc cluster. When we were satisfied with the performance of the new scheduler, we spun up another larger consolidated ad-hoc cluster with the new scheduler (also shown by the orange line), and all new ad-hoc Genie jobs were now routed to this latest incarnation. The two older clusters were terminated once all running jobs were finished (we call this a “red-black” push).

Summary

Even though Genie is now open source, and has been running in production at Netflix for months, it is still a work in progress. We think of the initial release as version 0. The data model for the services is fairly generic, but definitely biased towards running at Netflix, and in the cloud. We hope for community feedback and contributions to broaden its applicability, and enhance its capabilities.

We will be presenting Genie at the 2013 Hadoop Summit during our session titled “Genie - Hadoop Platform as a Service at Netflix”, and demoing Genie and other tools that are part of the Netflix Hadoop toolkit at the Netflix Booth. Please join us for the presentation, and/or feel free to stop by the booth, chat with the team, and provide feedback.

If you are interested in working on great open source software in the areas of big data and cloud computing, please take a look at jobs.netflix.com for current openings!

References

Genie OSS
Genie Wiki: Getting Started
Netflix Open Source Projects
@NetflixOSS Twitter Feed

• SHA1, SHA224, SHA256, SHA384, SHA512: digest
• HMAC SHA: sign, verify, importKey, exportKey, generateKey
• AES-128 CBC w/ PKCS#5 padding: encrypt, decrypt, importKey, exportKey, generateKey
• RSASSA-PKCS1-v1_5: sign, verify, importKey, generateKey
• RSAES-PKCS1-v1_5: encrypt, decrypt, importKey, exportKey, generateKey
• Diffie-Hellman: generateKey, deriveKey
• RSA-OAEP: wrapKey*, unwrapKey*
• AES-KW: wrapKey*, unwrapKey*

• Pytheas - A web based framework for quickly building dashboards
• Conformity Monkey - Maintain best practices for cloud deployments
• Zuul - Edge tier for dynamic filtering of requests
• Ice - AWS usage and cost analysis tool
• Genie - Hadoop platform abstraction service for EMR
• Lipstick - Visualization of Pig workflows

• Eucalyptus - V3.3 is now in production with support for NetflixOSS tools
• IBM - Scalable implementation of Acme Air demo using NetflixOSS
• Paypal - Rewrite of Asgard console to support Openstack deployments
• Riot Games - Cloud Native architecture based on many NetflixOSS projects

Amazon's Simple Workflow Service

Amazon's Flow Framework

@Workflow
@WorkflowRegistrationOptions(
    defaultExecutionStartToCloseTimeoutSeconds = 60L)
interfaceTestWorkflow {
@Execute(version = '1.0')
void doIt()
}

@Activities(version = '1.0')
@ActivityRegistrationOptions(
    defaultTaskScheduleToStartTimeoutSeconds = -1L,
    defaultTaskStartToCloseTimeoutSeconds = 300L)
interfaceTestActivities {
    String doSomething()
void consumeSomething(String thing)
}

interfaceTestActivitiesClient {
    Promise<String> doSomething()
void consumeSomething(Promise<String> thing)
}

classTestWorkflowImplimplementsTestWorkflow {
privatefinal TestActivitiesClient client = new TestActivitiesClientImpl();

void doIt() {
        Promise<String> result = client.doSomething()
  waitForSomething(result)
    }

@Asynchronous
void waitForSomething(Promise<String> something) {
        client.consumeSomething(something)
    }
}

Netflix OSS Glisten

classTestWorkflowImplimplementsTestWorkflow {
@Delegate
    WorkflowOperations<TestActivities> workflowOperations = SwfWorkflowOperations.of(TestActivities)

void doIt() {
        waitFor(activities.doSomething()) {
            activities.consumeSomething(it)
        }
    }
}

by Kristofer Baxter

In the past we’ve written about HTML5 Video (HTML5 Video in IE11 on Windows 8.1, and HTML5 Video at Netflix) but we haven't spoken much about how we built the player UI. The UI Engineering team here at Netflix has been supporting HTML5 based playback for a little over a year, and now seems like the right time to discuss some of the strategies and techniques we are using to support video playback without a plugin.

One of our main objectives is to keep Netflix familiar to our members. That means we’re keeping the design of the HTML5 player consistent with our Silverlight experience. Features should be rolled out simultaneously for the two platforms. However, HTML5 users will enter playback faster, can enjoy 1080p content when GPU accelerated, and keep all the functionality they know and love.

Silverlight UI	HTML5 UI

In order to achieve a similar look and feel, we needed to recreate a few key elements of the Silverlight UI:

1. Scale interface to users resolution
2. Minimize Startup time via minimal dependency on data
3. Ensure High Performance on low end hardware

Scaling interface to users resolution

No matter what resolution the browser window used for playback is, our current playback UI ensures all of the controls maintain the same percentage size on screen. This lets users choose their own dimensions for playing content without the UI getting in the way.

Normally, a modern web application could implement this using CSS vw and vh units. However, we found this approach to be inadequate for our needs. Our player can be displayed in two fashions -- taking over the entire initial containing block of a viewport, or a smaller portion. To solve for this, we implemented a sizing scheme based entirely on font-relative lengths.

In this small example, you can see the scaling implementation in a direct form.

<style>
.netflix-player-wrapper {
        font-size:16px;
}
#netflix-player {
        position: absolute;
        width:90%; height:90%;
        left:5%; top:5%;
        overflow: hidden;
        background:#ccc;
        font-size:1em;
}
#player-sizing {
        position: absolute;
        width:1em; height:1em;
        visibility: hidden;
        font-size:1em;
}
#ten-percent-height {
        position: absolute;
        width:80%; height:10em;
        left:10%; bottom:8em;
        background:#000;
        display: flex;
}
#ten-percent-height > p {
        display: block;
        margin:1em;
        font-size:2em;
        color:#fff;
}
</style>
<divclass="netflix-player-wrapper">
<divid="netflix-player">
<divid="player-sizing"></div>
<divid="ten-percent-height"><p>Text</p></div>
</div>
</div>
<script>
(function(){
var sizingEl = document.getElementById("player-sizing"),
        controlWrapperEl = document.getElementById('netflix-player'),
        currentEmSize =1.0;

function resize(){
var wrapperHeight = controlWrapperEl.getBoundingClientRect().height,
            sizingHeight = sizingEl.getBoundingClientRect().height,
            wrapperOnePercentHeight = wrapperHeight /100,
            offsetSize;

if(sizingHeight > wrapperOnePercentHeight){
            offsetSize = sizingHeight / wrapperOnePercentHeight;
            currentEmSize = currentEmSize / offsetSize;
}elseif(wrapperOnePercentHeight > sizingHeight){
            offsetSize = wrapperOnePercentHeight / sizingHeight;
            currentEmSize = currentEmSize * offsetSize;
}
        controlWrapperEl.style.fontSize = currentEmSize +"em";
}

    window.addEventListener("resize", resize,false);
    resize();
})();
</script>

We implement this resizing functionality on a debounced interval in the player UI. Triggering it on every window resize would be wasteful.

By making an em unit represent 1% height of the "netflix-player" container, we can size all of our onscreen elements in a scaling manner - no matter how or where the netflix-player container is placed in the document.

Minimize Startup time via minimal dependency on data

Browser plugins like Flash and Silverlight can take several seconds to initialize, especially on a freshly booted machine. Now that we no longer need to initialize a plugin to play content, we can begin playback faster. However, we learned a lot about quick video startup in Silverlight, and can borrow techniques we developed to make our HTML5 UI launch content even faster.

When possible, allow playback to begin without title metadata.

If we already know which title the customer has selected to play (like a specific episode or movie), we can just start playback of that title immediately. Once the user has begun to buffer content, the UI can request display metadata. Metadata for the player can be a large payload since it includes episode data (title, synopsis, predicted rating), and is personalized to the user. By delaying the retrieval of metadata, users begin streaming 500 to 1200ms sooner in real-world usage.

For other conditions, such when a customer clicks play on a TV show and we want to start playback at the last episode that they were watching, we retrieve the specific episode the user wants before starting the playback process.

Populate controls which depend on rich data as that data becomes available.

Since we can begin playback before the player UI knows anything except which title to play, the player UI needs to be resilient against missing metadata. We display a minimal number of controls while this data is being requested. These controls include play/pause, exit playback, and full-screen toggling.

We use an eventing framework to let individual components know when data state has changed, so each component can stay decoupled. Here’s an example showing how we handle an event telling us the metadata is now loaded for the title.

function populateStatus(){
if(Metadata.videoIsKnown(ObjectPool.videoId())){
// Update Status to reflect current playing item.
}else{
// Hide or remove current status
}
}

Metadata.addEventListener(Metadata.knownEvents.METADATA_LOADED, populateStatus);

Ensure High Performance on all hardware

Not everyone has the latest and greatest hardware at their disposal, but this shouldn't prevent all sorts of devices from playing content on Netflix. To this end, we develop using a wide variety of hardware and test using a wide range of representative devices.

We’ve found the issues preventing great performance on low end hardware can mostly be avoided by adhering to the following best practices:

Avoid repaints and reflows whenever possible.

Reflows and repaints while playing content is quite costly to overall performance and battery life. As a result, we batch reads and writes to the DOM wherever possible. This helps us avoid accidental reflows.

Take advantage of getBoundingClientRect to determine the size of object.

This is a very fast way to get the dimensions of an object. However, it isn’t a free operation and results should be cached whenever possible.

Caching the size of objects when dragging, instead of recalculating them every time they are needed, is one such way to reduce the number of calls in quick succession.

function setupPointerData(e){
    pointerEventData.dimensions ={
        handleEl:  handleEl.getBoundingClientRect(),
        wrapperEl: wrapperEl.getBoundingClientRect()
};
    pointerEventData.drag ={
        start:{ value: currentValue, max: currentMax },
        pointer:{ x: e.pageX, y: e.pageY }
};
}

function pointerDownHandler(e){
if(handleEl.contains(e.target)){
if(!dragging){
            setupPointerData(e);
            dragging =true;
}
}
}

function pointerMoveHandler(e){
if(dragging && isValidEventLocation(e)){
if(!pointerEventData ||!pointerEventData.dimensions){
            setupPointerData(e);
}
// Use the handleEl dimensions, wrapperEl dimensions, 
// and the event values to change the DOM.
}
}

We have a lot of work planned

We’re working on exciting new features and constantly improving our HTML5 Video UI, and we’re looking for help. Our growing team is looking for experts to join us. If you’d like to apply, take a look here.

• Release - on a Weekly deployment schedule
• Prod - on a Daily deployment schedule

[6] If the change was submitted to the Prod branch or a manual push trigger was invoked, the build is deployed to the canary cluster as well. It is worth mentioning that our unit of deployment is actually an Amazon Machine Image (ami), which is generated from a successful build. For the purposes of this post though, we'll stick to the term “build”.

Dependency Management

This is a good time to take a deeper look at our approach to dependency management. In the Netflix SOA implementation, services expose their interfaces to consumers through client libraries that define the contract between the two services. Those client libraries get incorporated into the consuming service, introducingbuild and runtime dependencies for the service. This is highly magnified in the case of the API because of its position in the middle of the hourglass, as depicted at the beginning of the post. With dozens of dependencies and hundreds of client libraries, getting a fully functional version of the API server stood up can in itself be tricky. To add to that, at any point in time, several of these libraries are being updated to support critical new functionality or bug fixes. In order to keep up with the volume and rate of change, we have a separate instance of our pipeline dedicated towards library integration. This integration pipeline builds an API server with the latest releases of all client libraries and runs a suite of functional tests against it. If that succeeds, it generates a configuration file with hardcoded library versions and commits it into SCM. Our developers and the primary build pipeline use this locked set of libraries, which allows us to insulate the team from instabilities introduced by client library changes but still keep up with library updates.

Testing

Next, we explore our test strategy in detail. This is covered in steps 1-4 in the pipeline diagram above. It is helpful to look at our testing in terms of the Software Testing Pyramid to get a sense for our multi-layered approach.

Local Development/Test iterations

Developers have the ability to run unit and functional tests against a local server to validate their changes. Tests can be run at a much more granular level than on the CI server; so as to keep the run times short and support iterative development. 

Continuous Integration

Once code has been checked in, the CI servers take over. The goal is to assess the health of the branch in the SCM repository with every commit, including automated code merges between various branches as well as dependency updates. The inability to generate a deployable artifact or failed unit tests cause the build to fail and block the pipeline.

Contract validations

Our interface to our consumers is via a Java API that uses the Functional Reactive model implemented by RxJava. The API is consumed by Groovy scripts that our UI teams develop. These scripts are compiled dynamically at runtime within an API server, which is when an incompatible change to the API would be detected. Our build process includes a validation step based on API signature checks to catch such breakages earlier in the cycle. Any non-additive changes to the API as compared to the version that’s currently in Production fail the build and halt the pipeline.

Integration Tests

A build that has passed unit and contract validation tests gets deployed to the CI server where we test it for readiness to serve traffic. After the server is up and running with the latest build, we run functional tests against it. Given our position in the Netflix stack, any test against the API is by definition an integration test. Running integration tests reliably in a live, cloud-based, distributed environment poses some interesting challenges. For instance, each test request results in multiple network calls on the API server, which implies variability in response times. This makes tests susceptible to timing related failures. Further, the API is designed to degrade gracefully by providing partial or fallback responses in case of a failure or latency in a backend service. So responses are not guaranteed to be static. Several backend services provide personalized data which is dynamic in nature. Factors such as these contribute towards making tests non-deterministic, which can quickly diminish the value of our test suites.

We follow a few guiding principles and techniques to keep the non-determinism under control.

• Focus on the structure rather than the content of the responses from the API. We leave content verification to the services that are responsible for generating the data. Tests are designed with this in mind.
• Account for cold caches and connections by priming them prior to test execution.
• Design the system to provide feedback when it is returning fallback responses or otherwise running in a degraded mode so tests can leverage that information.
• Quarantine unreliable tests promptly and automate the process of associating them to tasks in our bug tracking system .

These techniques have allowed us to keep the noise in the test results down to manageable levels. A test failure in our primary suite blocks the build from getting deployed anywhere.  

Another key check we perform at this stage is related to server startup time, which we track across builds. If this metric crosses a certain threshold, the pipeline is halted.

User Acceptance Tests

If a build has made it to this point, it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments. 

Canary Analysis

This is the final step in the process before the code is deployed globally. A build that has reached this step is a candidate for Production deployment from a functional standpoint. The canary process analyzes the readiness of the build from a systems perspective. It is described in detail in the post on Deploying the API.

Insight - tracking changes

Continuous Delivery is all about feedback loops. As a change makes its way through the pipeline, the automation delivers the results of each step to the concerned person(s) in real time via email and/or chat notifications. Build and Test results are also captured and made available on our Build Status Dashboard for later viewing. Everyone committing a change to the Release or Prod branch is responsible for ensuring that the branch is in a deployable state after their change. The Build Dash always displays the latest status of each branch, including a history of recent commits and test runs. Members of our Delivery Team are on a weekly rotation to ensure smooth operation of the pipelines.

Conclusion

The work we’ve done so far has been helpful in increasing our agility and our ability to scale with the growth of the business. But as Netflix grows and our systems evolve, there are newer challenges to overcome . We are experimenting in different areas like developer productivity, improved insight into code usage and optimizing the pipeline for speed. If any of these sound exciting, take a look at our open roles.