Talk

Kontextfrei: A new approach to testable Spark applications

Apache Spark has become the de-facto standard for writing big data processing pipelines. While the business logic of Spark applications is often at least as complex as what we have been dealing with in a pre-big data world, enabling developers to write comprehensive, fast unit test suites has not been a priority in the design of Spark. The main problem is that you cannot test your code without at least running a local SparkContext. These tests are not really unit tests, and they are too slow for pursuing a test-driven development approach. In this talk, I will introduce thekontextfrei library, which aims to liberate you from the chains of the SparkContext. I will show how it helps restoring the fast feedback loop we are taking for granted. In addition, I will explain how kontextfrei is implemented and discuss some of the design decisions made and look at alternative approaches and current limitations.

Date
2017-04-08
Time
15:00 - 15:30
Conference / Event
Scalar 2017
Venue
Museum of the History of Polish Jews, Warsaw
  • Slide 1
    1/32
  • Slide 2
    2/32
  • Slide 3
    3/32
  • Slide 4
    4/32
  • Slide 5
    5/32
  • Slide 6
    6/32
  • Slide 7
    7/32
  • Slide 8
    8/32
  • Slide 9
    9/32
  • Slide 10
    10/32
  • Slide 11
    11/32
  • Slide 12
    12/32
  • Slide 13
    13/32
  • Slide 14
    14/32
  • Slide 15
    15/32
  • Slide 16
    16/32
  • Slide 17
    17/32
  • Slide 18
    18/32
  • Slide 19
    19/32
  • Slide 20
    20/32
  • Slide 21
    21/32
  • Slide 22
    22/32
  • Slide 23
    23/32
  • Slide 24
    24/32
  • Slide 25
    25/32
  • Slide 26
    26/32
  • Slide 27
    27/32
  • Slide 28
    28/32
  • Slide 29
    29/32
  • Slide 30
    30/32
  • Slide 31
    31/32
  • Slide 32
    32/32

Swipe for more

Scroll or use your arrow keys for more

Scroll for more

Use your arrow keys for more

 Download Slides

TAGS