Week 3

Goals

• Exercise 1

  • Recognize significance of memory layout & data structures

  • Optimize serial code

• Exercise 2

  • Recognize pros/cons of different file formats

• Class Project

  • Apply "Programming in the Large" to your class project

  • Choose appropriate file format for a project

  • Choose appropriate memory layout & data structures for a project

Lessons along the way

• Big-O notation

• File formats: CSV, FITS, HDF5, JLD2

• Calling Python from Julia

Lab

Lab 3: Memory Access, Disk I/O, DataFrames (due Sept 19)

• Exercise 1: Dense Matrix-Vector Multiply: Optimizing serial code, Memory access paterns, Benchmarking

• Exercise 2: Benchmarking File I/O (& Calling Python Packages)

Readings

• Writing Scientific Software Ch 7: Software Design (26pg, ok to skip 7.11-12)

• A beginner's guide to Big-O notation

• Big-O Cheat Sheet (a useful reference, not to read end-to-end)

Additional Resources

• Week 3 Class Discussion: Priorities for Scientific Computing: Correctness, Documentation

• File Formats

  • FITSIO library written in C

  • HDF5

  • Apache Arrow

• Julia packages for reading files

  • Julia's FileIO.jl high-level API

  • Julia's JLD2.jl package

  • Julia's HDF5.jl package

  • Julia's FITSIO.jl package

  • Julia's Apache Arrow.jl implementation

  • Feather.jl

  • PyCall.jl documentation.

• Querying DataFrames

  • Query.jl

  • DataFramesMeta.jl)

  • JuliaDB.jl

• Miscelaneous

  • Regular expressions in julia

  • Astropy

• Julia

  • Julia Manual

  • Think Julia: How to Think Like a Computer Scientist

  • Learn Julia in Y Minutes