Vortrag

Kontextfrei: A new approach to testable Spark applications

Apache Spark has become the de-facto standard for writing big data processing pipelines. While the business logic of Spark applications is often at least as complex as what we have been dealing with in a pre-big data world, enabling developers to write comprehensive, fast unit test suites has not been a priority in the design of Spark. The main problem is that you cannot test your code without at least running a local SparkContext. These tests are not really unit tests, and they are too slow for pursuing a test-driven development approach. In this talk, I will introduce thekontextfrei library, which aims to liberate you from the chains of the SparkContext. I will show how it helps restoring the fast feedback loop we are taking for granted. In addition, I will explain how kontextfrei is implemented and discuss some of the design decisions made and look at alternative approaches and current limitations.

Datum
08.04.2017
Uhrzeit
15:00 - 15:30
Konferenz / Veranstaltung
Scalar 2017
Ort
Museum of the History of Polish Jews, Warsaw
  • Folie 1
    1/32
  • Folie 2
    2/32
  • Folie 3
    3/32
  • Folie 4
    4/32
  • Folie 5
    5/32
  • Folie 6
    6/32
  • Folie 7
    7/32
  • Folie 8
    8/32
  • Folie 9
    9/32
  • Folie 10
    10/32
  • Folie 11
    11/32
  • Folie 12
    12/32
  • Folie 13
    13/32
  • Folie 14
    14/32
  • Folie 15
    15/32
  • Folie 16
    16/32
  • Folie 17
    17/32
  • Folie 18
    18/32
  • Folie 19
    19/32
  • Folie 20
    20/32
  • Folie 21
    21/32
  • Folie 22
    22/32
  • Folie 23
    23/32
  • Folie 24
    24/32
  • Folie 25
    25/32
  • Folie 26
    26/32
  • Folie 27
    27/32
  • Folie 28
    28/32
  • Folie 29
    29/32
  • Folie 30
    30/32
  • Folie 31
    31/32
  • Folie 32
    32/32

Nach links wischen, um ggf. weitere Folien zu sehen

Scrollen oder die Pfeiltasten nutzen, um ggf. weitere Folien zu sehen

Scrollen, um ggf. weitere Folien zu sehen

Die Pfeiltasten nutzen, um ggf. weitere Folien zu sehen

Folien downloaden

TAGS